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Executive summary 


The project 


The Evidence-based Literacy Support—‘Literacy Octopus’ Trial tested a range of dissemination 
interventions and resources, all of which aimed to engage schools in using evidence-based materials 
to improve teaching and learning in Key Stage 2 literacy. Four delivery partners provided 
interventions. These included light-touch, ‘passive’ approaches—such as emailing materials to 
schools—and more ‘active’ support such as face-to-face events about putting research into practice. 
Separately, the ‘Literacy Octopus’ Dissemination Trial (a sister trial which did not include a support 
element) tested the impact of some of the former, more passive approaches in a large-scale trial. This 
report concerns a trial of the latter, more active approaches, but also included ‘passive’ elements. 


The delivery partners were the Institute for Effective Education (IEE) at the University of York, the 
Campaign for Learning in partnership with Train Visual, the Centre for Evaluation and Monitoring 
(CEM) at Durham University, and NatCen in collaboration with ResearchEd. The different 
interventions are outlined below: 


IEE Arm 1 Passive Regular printed and electronic materials, including the 
magazine Better Evidence-based Education and ‘Best 
Evidence in Brief’ email, and access to a searchable 
database, Evidence 4 Impact. 


Arm 2 Active Arm 1 plus invitation to an evidence-based literacy 
programmes event. 


Campaign Arm 3 Passive Access to Teaching How2s website, including evidence- 


for based visual guides on CPD. 

Learning/ 

Train Visual 

Arm 4 Active Arm 3 plus invitation to a one-day introduction to using the 
Teaching How2s website, and updates on using the visual 
guides. 

CEM Arm 5 Passive Hard copy evidence-based KS2 literacy teaching materials, 
including a teacher’s guide and monthly classroom activity 
posters. 

Arm 6 Active Arm 5 plus invited to one twilight CPD session. 
light 
Arm 7 Active Arm 6 plus invited to one further twilight CPD session, use of 


advanced | pupil diagnostic tools, and teacher peer observation 
between sessions. 


NatCen/ Arm 8 Passive Invitation to free Saturday conference on research evidence 
ResearchEd relating to Key Stage 2 literacy. 
Arm 9 Active Arm 8 plus invitation to two webinars to provide support on 


applying the research from the conference in their schools. 


The evaluation was designed to assess the impact of the different interventions on pupil outcomes 
and teachers’ use of, and engagement with, research. The interventions began in spring 2015 with a 
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range of materials and light-touch support offered through to the following academic year. Key Stage 
2 English results from the summer 2016 were used to assess the impact on pupils. This was a large- 
scale effectiveness trial. The evaluator, NFER, recruited 823 primary schools; 60 were allocated to 
each of the nine intervention arms and 283 to the control group. The qualitative evaluation included 
observations of activities, interviews, and case studies. The trial was funded by the Education 
Endowment Foundation (EEF), the Department for Education, and the Mayor's London Schools 
Excellence Fund as part of a round of funding exploring Research Use in Schools. 


Key conclusions 


The project found no evidence that any of the interventions improved pupils’ Key Stage 2 
English scores. The five padlock security rating means we have high confidence in this result. 


There was no evidence of impact on any of the six teacher Research Use Measures used in 
this trial. However, we have limited confidence in this result given the low response rate to the 
questionnaires designed to capture these outcomes, and some measures were only 
moderately reliable. 


Schools’ level of engagement varied: six out of ten schools did not engage to the level 
expected by the providers, although a small proportion engaged to a greater extent than 
expected (for example by hosting CPD sessions). Reasons for not engaging included lack of 
time, the timing and location of events, and a preference for face-to-face support rather than 


online or remote formats only. 


Teachers felt research evidence was most effectively communicated when it was interactive, 
accessible, relevant, included a balanced and credible discussion of the evidence, and focused 
on how to apply the evidence in practice. Where schools went on to implement changes in light 
of Literacy Octopus engagement, these came about through mechanisms such as in-school 
collaboration, further enquiry, and trying out, reviewing, adapting, and embedding approaches. 


The lack of impact across the different interventions suggests that simply communicating 
research evidence to schools is not enough to improve outcomes. How easily the presented 
evidence can be used in practice—and the conditions in schools for implementing evidence- 
based change—might be just as important. Further research should assess whether 
interventions can transform evidence into practical action, and develop supportive 
implementation conditions in schools. 


EEF security rating 


These findings have very high security. The trial was a large effectiveness trial which tested whether 
the interventions worked in large numbers of schools. It was a well-powered, school-level, 
randomised controlled trial involving 823 primary schools. Pupils in schools in each intervention arm 
were similar to those in control schools. The primary outcome—pupil attainment in Key Stage 2 
literacy—was based on data accessed from the National Pupil Database. Only four schools’ data was 
not used in the final analysis due to being unable to match these four schools to the NPD data. In the 
remaining schools, 6% of pupils did not have test scores for both Key Stage 1 and Key Stage 2 but 
this did not affect balance at baseline. 


Additional findings 


There was no evidence that any of the interventions had an impact on pupils’ English results at KS2 
(2015/2016). This was also true when considering only pupils who have ever been eligible for free 
school meals (FSM). Schools’ different levels of engagement with the Literacy Octopus interventions 
made no difference to these results. Literacy Octopus interventions made no difference to teachers’ 
research use as measured by a validated Research Use Outcomes Survey: when each intervention 
arm was compared to the control group there was no evidence of any impact on any of the six 
Research Use Measures. 
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Monitoring data showed that schools’ levels of engagement with the interventions were relatively low: 
just over four in ten of the 540 intervention schools did not engage, and only four in ten engaged as 
much as, or more than, was expected. The other schools engaged a little, though less than expected. 
Teachers engaged where they felt the materials or support were communicated in an accessible way 
(such as posters that were easy to display, or easy event location) and that the content of materials 
was relevant (for example, linked to an existing need). The passive or active nature of support did not 
appear to be the key determinant in whether schools engaged or took further action. 


Where schools went on to implement changes in light of Literacy Octopus engagement, these came 
about through mechanisms such as in-school collaboration, schools undertaking further enquiry or 
follow-up activity with the provider, schools trying out, reviewing, adapting, and embedding 
approaches, and having a planned approach to implementation. In addition, having a ‘research 
champion’ to drive the change was key in case-study schools. Where teachers did engage, most 
perceived that participating in the Literacy Octopus had led to positive impacts on their practice 
enabling them to reflect on, reinforce, and change their classroom practice, as well as discuss best 
practice with colleagues in their school. 


Literacy Octopus interventions were all reasonably light touch, and as, overall, these had no impact 
on the primary and secondary outcomes measured in this trial, providers of research evidence need 
to consider the findings from the Implementation and Process Evaluation of this study. This 
highlighted the need to focus on transforming the research so that it can easily be applied in practice, 
and to consider implementation factors such as the conditions within the school for engaging with 
evidence and adopting new approaches. This recommendation fits with the findings from the ‘Literacy 
Octopus’ Dissemination Trial which showed that simply disseminating research evidence to large 
numbers of schools does not impact on attainment. 


Cost 


All interventions were very low cost for schools—between £0.09 and £10.77 per pupil per year over 
three years. This included real-world costs relating to photocopying materials, subscriptions, event 
fees, and so on. School and teacher time (rather than monetary cost) was not measured, but was a 
challenge for some of the schools involved—both in terms of initial engagement and time to 
implement evidence-based strategies. 
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Table 1: Summary of impact on primary outcome (Key Stage 2 English) 


IEE Passive 
Arm 1 vs control 
IEE Active 
Arm 2 vs control 
How2s Passive 
Arm 3 vs control 
How2s Active 
Arm 4 vs control 
CEM Passive 
Arm 5 vs control 
CEM Active Light 
Arm 6 vs control 
CEM Active 
Arm 7 vs control 
ResearchEd 
Passive 
Arm 8 vs control 
ResearchEd Active 
Arm 9 vs control 
IEE Passive Arm 1 
everFSM vs control 
everFSM 
IEE Active Arm 2 
everFSM vs 
everFSM control 
How2s Passive Arm 
3 everFSM vs 
everFSM control 
How2s Active Arm 
4 everFSM vs 
everFSM control 
CEM Passive Arm 5 
everFSM vs 


everFSM control 


CEM Active Light 
Arm 6 everFSM vs 
everFSM control 
CEM Active Arm 7 
everFSM vs 
everFSM control 
ResearchEd 
Passive Arm 8 
everFSM vs 
everFSM control 
ResearchEd Active 
Arm 9 everFSM vs 
everFSM control 


-0.02 
(-0.14, 0.10) 
-0.04 
(-0.15, 0.08) 
0.00 
(-0.12, 0.11) 
-0.03 
(-0.14, 0.09) 
0.00 
(-0.12, 0.11) 
0.03 
(-0.09, 0.14) 
0.03 
(-0.09, 0.14) 


0.00 
(-0.11, 0.12) 


0.01 
(-0.11, 0.13) 


-0.03 
(-0.19, 0.12) 


-0.05 
(-0.19, 0.10) 


-0.07 
(-0.22, 0.08) 


-0.01 
(-0.15, 0.14) 


-0.02 
(-0.17, 0.14) 


0.05 
(-0.11, 0.20) 


0.04 
(-0.11, 0.19) 


0.04 
(-0.11, 0.18) 


0.02 
(-0.13, 0.18) 


£0.73 


£3.90 


£4.11 


£0.09 


£0.39 


£10.77 


£0.26 


£0.26 


2,203 
2,337 
2,448 
2,174 
2,080 


2,386 


2,474 


2,122 


526 


537 


0.98 


88888 


84888 


88888 


88888 
88888 


886888 exeee | 
88888 eeeee | 
886888 exeee | 


LEELE 


LEELE 


LEELE 


LEELEL 


Eee de ae 


* ‘Number of pupils’ refers to the number of pupils’ results analysed in the primary outcome multi-level model. 
Note, the model included results from 10,280 control group pupils. The everFSM model contained results from 
3,094 control group pupils. 
*“The p-value results from a single likelihood ratio test (LRT) across all trial arms. The hypothesis for this test was 
that there is no difference in mean score, adjusted for baseline, between each arm and control. 
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Introduction 


Interventions 


The Evidence-based Literacy Support — the ‘Literacy Octopus’ Trial was a multi-armed trial 
commissioned by the Education Endowment Foundation (EEF) focused on a range of different 
methods of communicating and disseminating evidence-based materials and support to schools. The 
trial set out to measure their effect on pupils’ Key Stage 2 (KS2) literacy attainment and on teachers’ 
research use and engagement. Interventions were organised by four separate providers brought 
together under a common protocol to form a multi-armed trial. As discussed in the section on 
background evidence, there is a range of both ‘push’ and more ‘transformational’ approaches to 
supporting research engagement in schools. Each provider communicated their materials to schools 
using both approaches. ‘Push methods’ (known as the ‘passive intervention arms’) involved 
information being sent to schools by various means. The ‘supported’ or ‘transformed methods’ (known 
as the ‘active intervention arms’) involved the same information being sent to schools (as in the 
respective passive arm) but with additional support such as continuing professional development 
(CPD) or webinars. 


In total, the trial consisted of ten arms (nine intervention arms and one control arm). Three of the 
Literacy Octopus providers each offered two interventions (active and passive); one of the providers 
offered three interventions (active, active light, and passive—a design feature commissioned by the 
EEF in order to allow this provider to trial three levels of support). The providers were asked to focus 
on literacy in KS2, and were also asked to include an element of co-operative learning in their 
materials to ensure a common theme across the provision. Intervention activities took place from 
March 2015 to July 2015. Active support sessions were provided within this period in locations in the 
north and south of England appropriate to the geographical randomisation allocation of schools (see 
Methods: Randomisation). Where interventions involved sending materials to schools, providers 
continued to disseminate their materials to their Literacy Octopus schools during the following 
academic year (September 2015 to July 2016). 


Each intervention had a slightly different Theory of Change (ToC). Some aimed to directly engage 
teachers with evidence-based teaching methods; some aimed to influence school-leaders and school- 
level decisions about taking on evidence-based interventions; and others aimed to encourage peer-to- 
peer discussion about research findings. All of the interventions hypothesised that active intervention 
alongside the materials would support greater changes in practice. The ToCs for each provider are 
presented in Appendix D. 


The IEE interventions 


The Institute for Effective Education at the University of York! provides printed and electronic 
materials for schools focusing on educational research evidence and recent research findings. Their 
interventions for the Literacy Octopus involved: 


e Passive intervention (Arm 1): schools received regular printed and electronic materials from 
February 2015 to July 2016, including regular issues of the IEE magazine Better Evidence- 
based Education? (which brings together evidence on particular topics, and included some 
literacy-specific volumes for schools taking part in the Literacy Octopus), its usual fortnightly 
email, ‘Best Evidence in Brief’, summarising new research findings and containing links to 
relevant articles, and access to Evidence 4 Impact, a searchable database of evidence-based 
education intervention programmes. (Neither ‘Best Evidence in Brief’ nor Evidence 4 Impact 
were specifically adapted for the Literacy Octopus.) 


' The Institute for Effective Education (IEE) at the University of York: https://the-iee.org.uk/ 
? Better Evidence Based Education: http://www.betterevidence.org/ 
3 Best Evidence in Brief: http://www.beib.org.uk/ 
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e Active intervention (Arm 2): as well as receiving the same materials as schools in the 
passive arm, teachers from schools in the active arm were invited to attend a one-day 
evidence fair to hear about particular literacy interventions from developers of evidence-based 
programmes and from schools that had used the interventions. The evidence fairs took place 
in April 2015 in York for schools in the north of England, and in May 2015 in London for 
schools in the south of England. Places were offered to several senior leaders from each 
school. Pre- and post-conference support was also offered to schools taking part in this arm. 


Teaching How2s interventions 


Teaching How2s?4 runs a subscription website designed for school and further education college CPD. 
Schools or colleges that subscribe to the website gain access to visual guides for particular teaching 
and CPD techniques. The focus is on supporting teachers to implement evidence-based techniques 
through step-by-step guides, with theoretical underpinnings available separately. The arms for this 
trial were managed by the Campaign for Learning on behalf of Train Visual (the organisation that 
owns Teaching How2s). The existing Teaching How2s were updated to include additional content 
specifically applicable to primary schools and literacy. 


e Passive intervention (Arm 3): schools were given a free subscription to the Teaching How2s 
website from February 2015 to July 2016 and were able to access all content, including 
content developed for the Literacy Octopus including an online learning centre. 


e Active intervention (Arm 4): as well as the above, a senior leader from each school in the 
active arm was invited to a one-day event introducing the website. The day included a 
presentation by a school already subscribing to the service. Schools in the north were invited 
to a one-day session in Manchester in February 2015; schools in the south were invited to a 
one-day session in London in February 2015. In addition, schools received updates that 
supported schools to use the visual guides. This included blog posts that showcased how 
project schools were using How2s for their professional development. 


CEM 


The Centre for Evaluation and Monitoring at Durham University® developed evidence-based materials 
which focused on strategies for teaching literacy at Key Stage 2. The materials included a booklet, 
Improving Reading: A Guide for Teachers, a ‘Top Tips’ card (to act as a cue to the booklet), a 
staffroom poster (to act as a cue to the evidence-based materials and increase awareness of the 
project throughout the school), and monthly classroom activity display posters. Some schools were 
also offered additional CPD sessions and a diagnostic assessment for pupils. Their Literacy Octopus 
interventions involved: 


e Passive intervention (Arm 5): in March 2015, schools received the evidence-based 
materials, including the Improving Reading: A Guide for Teachers, a ‘Top Tips’ card, a 
staffroom poster, and monthly classroom posters. 


e Active intervention (light touch, Arm 6): in addition to the evidence-based materials, 
schools in the light-touch active arm were invited to a twilight CPD session where they were 
guided by a CEM researcher through the booklet and its evidence base. Twilight CPD 
sessions took place from March to April 2015 in a number of locations in the north and south 
of England. Around five to eight schools were invited to each session in anticipation of two to 
four teachers or senior leaders attending from each school. 


e Active intervention (Arm 7): schools in this active arm received the materials and were also 
invited to the first twilight CPD session alongside schools in the active light-touch arm. 


4 Teaching How2s: https://teachinghow2s.com/ 
5 The Centre for Evaluation and Monitoring (CEM) at Durham University: http:/Awww.cem.org/ 
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Following the first CPD session, schools were also given access to a computer-based 
diagnostic profiling tool (INCAS) of reading and spelling to enable KS2 teachers to pinpoint 
pupils’ individual literacy strengths and weaknesses. These schools were offered a further 
CPD session in May and June 2015 to discuss the strategies in Improving Reading: A Guide 
for Teachers in relation to the profiles of their pupils. Teachers were also expected to 
undertake peer-learning observations in between the two CPD sessions. 


ResearchEd and NatCen interventions 


ResearchEd® and NatCen worked together to provide schools with a one-day conference, using the 
ResearchEd approach, on research evidence for Key Stage 2 literacy. The conference was held on a 
Saturday, was free of charge for participants, and involved presentations from a number of high- 
profile academics and practitioners. 


e Passive intervention (Arm 8): individual teachers and senior leaders from schools received 
an invitation to the literacy conference in April 2015 (held in York for schools in the north of 
England and in London for schools in the south of England). 


e Active intervention (Arm 9): as well as receiving the invitation to the literacy conference, 
schools in the active arm were offered the opportunity to become involved in two online 
webinars: the first supported them to prepare for the conference and the second encouraged 
them to use the research-based practice in their schools. 


Conirol group 


There was a single control group (known as Arm 10) for all the interventions. Schools in the control 
group carried on ‘business as usual’, although they were occasionally approached by members of the 
evaluation team to provide information on their current levels of research engagement and their KS2 
literacy activities. 


The section on Process Evaluation: Fidelity provides information on any changes made by providers 
during implementation to these intended plans. 


Background evidence 


Historically, the education sector has lagged behind other professions, such as medicine and social 
care, in being considered as evidence-informed (Nutley ef a/., 2007 cited in Nelson and O’Beirne, 
2014; Hargreaves, 1996 cited in Nelson and O’Beirne, 2014). In 1998, a review of educational 
research in England concluded that there was little evidence of the use and impact of research on 
educational policy or practice (Hillage et a/., 1998). More recently, authors in the field continue to 
highlight an enduring misalignment in the production of evidence and its use in policy and practice 
(Becheikh et al., 2009; Nutley et al., 2007 cited in Griggs et a/., 2016; DfE’s March 2016 White Paper; 
Greany, 2015). In 2013, less than half of a sample of 1,577 teachers and school leaders reported 
using research evidence to inform decision-making about approaches to improve pupil outcomes 
(Ager and Pyle, 2013). 


A range of challenges has been noted in relation to research-use in education practice. These 
include: the scale and generalizability of research findings, the often inaccessible nature of findings to 
a non-academic audience, practitioners’ scepticism of the value of research evidence, and teachers’ 
capacity for interpreting findings (CUREE, 2011; Goldacre, 2013; Becheikh et al., 2009; Durbin and 
Nelson, 2014; See et al, 2016). The Department for Education’s March 2016 White Paper, 
‘Educational Excellence Everywhere’, outlines the challenges that teachers often face in accessing, 
interpreting, and translating research evidence to inform their teaching practice and highlights issues 
concerning the relevance of much research to the needs of teachers and schools. 


§ ResearchEd: http://www.researched.org.uk/ 
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Over the past two decades there has been a growing recognition of the value of research in informing 
understanding about what works in ensuring the best educational outcomes for young people. The 
government is keen to develop a more evidence-informed teaching profession and set out a number 
of plans to support this in the White Paper, ‘Educational Excellence Everywhere’ (2016). Goldacre 
(2013) asserted that ‘by collecting better evidence about what works best and establishing a culture 
where this evidence is used as a matter of routine, we can improve outcomes for children, and 
increase professional independence’ (p. 7). In an increasingly autonomous, decentralised, and self- 
improving school system, research evidence is becoming ever more important in informing local 
professional judgements to improve outcomes for children. There is thus mounting impetus to ensure 
greater alignment between the supply and use of research evidence. 


There is a range of approaches to supporting schools’ use of research: from ‘push’ approaches— 
disseminating and distributing research findings to schools—to more transformational and interactive 
forms involving, for example, practical guidance and participatory research. The former may be the 
most common and low cost approach, but evidence indicates that transformative and interactive 
approaches more effectively address the challenges that teachers face in using research evidence 
(Marshall and Drummond, 2006 cited in CUREE, 2011). Nelson and O’Beirne (2014) highlight the 
importance of both transforming research findings into teacher-specific resources and of social 
interaction between researchers and teachers to help mediate research messages and lead to the 
implementation of research-based strategies. For instance, Tymms and Merrell (2006) mediated 
research findings by producing a practical guidance booklet for schools on evidence-based strategies 
to address inattentive, hyperactive, and impulsive pupil behaviour that led to positive impacts on pupil 
behaviour and teacher morale. 


Becheikh et al. (2009) distinguished four models of knowledge transformation: diffusion and 
dissemination of research findings; using researchers to identify solutions to identified problems; 
translating evidence into useable guidance for practice; and researchers and teachers co-producing 
and using research knowledge. Sharples (2013, based on Shepherd 2007) suggested that effective 
mobilisation of evidence involves a cycle of ‘production’ by research organisations, ‘synthesis’ and 
‘transformation’ into accessible and useable outputs, and ‘implementation’ in policy and practice to 
develop pupil outcomes. Various strategies are aiming to enhance different aspects of this ‘eco- 
system’ to improve the synergy and interaction between these different aspects of the cycle. The 
government’s plans to build capacity for evidence-informed teaching (set out in the White Paper, 
‘Educational Excellence Everywhere’, 2016) include: a portal for teachers to post research questions 
and access educational journals; the expanding role of the Education Endowment Foundation (EEF) 
in exploring and sharing robust evidence on what works in education; and the establishment of an 
independent ‘College of Teaching’ which will encourage the use of evidence from research to inform 
practice and provide evidence-informed professional development for teachers. There is also greater 
emphasis on research producers and intermediary organisations synthesising research findings and 
transforming evidence into usable guidance for practice. 


Despite these efforts, there is insufficient understanding of how evidence is most effectively 
disseminated, communicated, and mobilised to influence practice—or of the relative effectiveness of 
different approaches to knowledge production, transformation, and implementation (Nelson and 
O’Beirne, 2014; Greany, 2015). The Literacy Octopus trial aims to shed light on this by exploring the 
effects and effectiveness of different approaches to communicating research findings to schools. The 
evaluation explores the impact of these approaches on teachers’ awareness and understanding of 
research and their subsequent action and practice. Existing evidence from studies of effective 
approaches to teachers’ CPD suggests that this may be a hierarchical process whereby awareness 
and understanding are necessary prerequisites for changes to action and practice (Harland and 
Kinder, 1997)—although some of the theories of change in the Literacy Octopus study challenge this 
notion (see Appendix D). 


Whilst all the dissemination approaches in this study could be described as reasonably ‘light touch’, 
the study design ensured a range of more- and less-supported activity in order to explore the 
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hypothesis that more ‘active’ approaches to communicating research findings to schools (for example, 
interactive forms of communication and knowledge-sharing and findings translated into guidance for 
practice) will be more effective than simple ‘passive’ approaches (such as sending a research 
summary document to schools). As discussed above, elsewhere in the literature this distinction has 
also been posited as being important for effective knowledge mobilisation (Becheikh et al., 2009; 
Durbin and Nelson, 2014; Nelson and O’Beirne, 2014; Sharples, 2013). The Literacy Octopus process 
evaluation also investigates examples of how research evidence is then implemented in schools to 
improve teachers’ practice and pupil outcomes providing valuable insights about how the mechanisms 
and conditions underpinning effective knowledge mobilisation can best be supported and recreated. 


Evaluation objectives 


Primary research question 


The primary research question for the impact evaluation was: What are the effects of different ways of 
communicating research evidence and findings to teachers and schools on pupil attainment (in terms 
of Key Stage 2 literacy for the 2016 Year 6 cohort)? 


Secondary research questions 


The main secondary research question for the impact evaluation was: What are the effects of different 
ways of communicating research evidence and findings to teachers and schools on pupil attainment 
(in terms of Key Stage 2 literacy for the 2015 and 2017 Year 6 cohorts)? 


The aim of this analysis was to measure the differential progress in literacy development between 
pupils in each arm of the trial in order to measure the effect different approaches to disseminating 
evidence-based materials have on literacy attainment over three Year 6 cohorts (those of 2014/2015, 
2015/2016, and 2016/2017 respectively). Analysis relating to the latter two academic years will be 
reported in the 2018 Literacy Octopus addendum report. 


Other secondary research questions for the impact evaluation were: What is the effectiveness of 
different communication approaches in terms of schools’ and teachers’ dispositions towards research, 
and to what degree does research information inform teaching and learning? (The latter measured by 
factor scores calculated from a standard Research Use Outcomes Survey; Poet et a/., 2015, 
unpublished.) This involved exploring impact in terms of teachers’ awareness, understanding, and 
actions (changes in practice) in relation to the evidence-based materials or activity they were 
allocated to. 


Other aims included the exploration of whether the level of school engagement in the Literacy 
Octopus interventions had any impact on the primary outcome (2016 cohort’s KS2 literacy attainment) 
using Complier Average Causal Effect (CACE) analysis.’ A further aim is to explore which aspects of 
research use, according to survey results, are associated with pupil attainment outcomes—that is, 
whether certain measures of research use within trial arms are associated with improved pupil 
attainment (this is outlined in the Path analysis in the SAP; see footnote 7). The Path analysis will be 
reported in the 2018 Literacy Octopus addendum report. 


Subgroup analysis 


Sub-group analysis on the primary outcome (2016 cohort KS2 literacy attainment) was explored for 
pupils who have ‘ever received free school meals’ (‘everFSM’). 


Process evaluation questions 


7 The statistical analysis plan (SAP) for this trial can be found at: 
https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Protocols/SAP_Literacy_Octopus 
-_Active.pdf 
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For the process evaluation, the main questions were: How are these outcomes achieved? What are 
the mechanisms that bring about improved research use for teachers and schools? What actions 
have they taken, and is this perceived to improve teaching and learning, and ultimately pupil 
outcomes? 


The evaluation objectives are set out in the published protocol: 


https ://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Protocols/EEF_Project 
_Protocol_ TheLiteracyOctopusActiveTrial.pdf. 


Ethical review 


This study obtained approval from NFER’s Code of Practice Group on 3 July 2014. This study 
planned to use de-identified pupil administrative data—anonymized Pupil Matching Reference 
(aPMR) from the Department for Education (DfE) National Pupil Database (NPD) in relation to Key 
Stage 1 and Key Stage 2 attainment data. There were no special ethical concerns regarding this; 
consent to access NPD was obtained during the recruitment process via the headteacher out of 
courtesy, although note that as anonymized pupil data would be used, data access permissions were 
not needed. 


The project team and Code of Practice Group gave specific consideration to the issues of recruiting 
schools to a range of possible activities—in particular, that some arms would require more resources 
in terms of teacher time and monetary cost than others. Where a conference fee was going to be 
charged (reflecting real-world conditions), the Group felt this was substantially over and above 
reasonable expectations for research participation, and the Literacy Octopus provider reduced the fee 
for this arm (a small fee was still charged as it was felt this was part of teachers’ valuing and securing 
their place on an event, as per the real world). Invitation materials were careful to set out the range of 
possible activities each school might be allocated to, from the most intensive (for example, CPD 
sessions and online help) to the least intensive, such as receiving regular mailings of new reports 
about teaching literacy. In addition, the dedicated NFER Literacy Octopus website® provided an 
overview of each of the ten arms. 


Schools opted into the trial by the headteacher (or their designated deputy) signing a memorandum of 
understanding (MoU) during recruitment. On this form, they also provided the name and job title of the 
member of staff allocated to be the main contact throughout the study. The information sheet, 
dedicated NFER Literacy Octopus website, invitation letter, reply form, and online surveys all 
contained relevant information about consent and how the survey data and NPD data would be used- 
—in particular, that teachers completing the survey consented for their contact details to be shared 
with their allocated delivery partner for the purposes of the project, and for their survey data to be 
used in analysis of DfE NPD data (the path analysis described in the evaluation aims above). It was 
also made clear that while the EEF and Fischer Family Trust data archives would contain NPD 
matched data, no individual school, teacher, or pupil would be identified in any report arising from this 
evaluation. 


Appendix C provides the project information sheet, recruitment invitation letter to schools, and school 
MoU and reply form. 


Project team 


The principal investigator was Dr Ben Styles, head of NFER’s Education Trials Unit. The day-to-day 
trial manager was Pippa Lord, Senior Research Manager in NFER’s Centre for Evaluation and 
Consultancy. They were supported by a team of researchers including: Adam Rabiasz and Palak Roy 
(project statisticians), Katherine Fowler and Jennie Harland (implementation and process evaluators), 


8 https://www.nfer.ac.uk/schools/taking-part-in-our-research/EEFA/ 
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Dave Hereward (survey operations lead) and in the recruitment stage, Dr Anneka Dawson (school 
recruitment lead, then NFER). 


The projects were delivered by four organisations or partnerships referred to as ‘Literacy Octopus 
providers’. These were: the Institute for Effective Education (IEE) at The University of York (in charge 
of delivering two arms); the Campaign for Learning on behalf of Train Visual (the organisation that 
owns Teaching How2s; two arms); the Centre for Evaluation and Monitoring (CEM) at Durham 
University (three arms); and NatCen working with ResearchEd (two arms). 


NFER was responsible for recruiting schools, overseeing the baseline survey, randomising schools, 
conducting the implementation and process evaluation, overseeing the outcomes survey, pupil data 
analysis and reporting. Literacy Octopus providers were responsible for all project delivery 
communications with schools, including initial welcome information, disseminating their materials to 
schools in the passive and active arms, and delivering support to schools in the active arms. NFER 
put in place a data sharing agreement with each provider to cover how the relevant allocated list of 
schools and teacher contact details would be shared (per arm), and how providers’ would share 
monitoring information with NFER. 


The project was supported and guided by EEF staff Eleanor Stringer and Professor Jonathan 
Sharples, joined, from 2016 onwards, by Dr. Anneka Dawson. 


Trial registration 


The Literacy Octopus Trial is registered at http://www.isrctn.com/ISRCTN84508118. This 
registration relates to both the dissemination and evidence-based literacy support trials — known in the 
protocol, statistical analysis plans and analysis syntax as the Passive and Active Trials respectively. 


Education Endowment Foundation 13 


Evidence-based Literacy Support — the ‘Literacy Octopus’ Trial 


Methods 


Trial design 


The Evidence-based Literacy Support — the ‘Literacy Octopus’ trial was commissioned as a multi- 
armed trial of four providers’ evidence-based literacy materials and support for schools. Schools were 
randomised into one of ten groups or arms—nine intervention arms and a ‘business as usual’ control 
arm. Each provider communicated evidence-based materials in two ways, passive and active, with 
each school in the trial only experiencing one of these means of intervention. Three of the four 
providers each delivered one passive and one active arm (IEE, Teaching How2s, and 
ResearchEd/NatCen). The fourth, CEM, delivered three arms (passive, active light, and active). 


Since many of the research communication strategies involved whole-school ‘receipt’ of materials 
(rather than specific or selected teacher involvement), school-level (rather than teacher-level) 
randomisation was employed. This included stratification by geographical area using a binary variable 
that indicated whether the school was situated in the north or south of England. This was a necessary 
part of the design as some interventions involved travel to set locations in the north (York) and south 
of England (London). 


The trial was designed so that each arm would be compared to the control group, that is, a many-to- 
one comparison design where each group mean is compared to the control group mean. No 
comparisons were planned between the trial’s intervention arms (see section on Analysis). 


Control schools continued ‘business as usual’ for the duration of the trial. They were not permitted to 
take part in, or access, the Literacy Octopus materials during the trial (although for some freely 
available resources this was impossible to guarantee) but could take part in other literacy 
improvement interventions or research-engagement activities as part of their normal school 
development plans. The process evaluation captured the range of ‘business as usual’ activities. 
Control group schools received a £30 book token or Amazon token as acknowledgement of their 
commitment to participate in the outcomes survey. All teachers completing the outcomes survey, in 
both intervention and control schools, were offered a £5 voucher or a donation to charity to 
acknowledge their contribution to the survey. 


Following publication of the trial protocol, a change to the Key Stage 2 NPD data source was made 
after advice from the DfE concerning the use of data obtained from the new KS2 tests. The DfE’s 
preference is for amended, rather than unamended, NPD data from these new assessments to be 
used in 2016. A protocol amendment was published in June 2016 to reflect this change in data source 
and timetable. The results from analysis of 2014/2015 KS2 data (the interim data for this trial) will be 
reported in the 2017/2018 addendum report. A small number of control group interviews were added 
to the process evaluation design in order to capture examples of any increased activity relating to 
research engagement—as it was felt this had become a rapidly growing environment during the 
course of the trial period. As an addition to the protocol, the teacher survey included some 
implementation and process evaluation (IPE) questions to ascertain views on the communication 
approaches, perceived outcomes, and formative recommendations. 


Participant selection 


The main sample consisted of English primary schools with a Year 6, excluding special schools, 
independent schools, and local authorities (LAs) that were classed as too far from York and London 
for travel within one day (which would be required for several of the arms of the intervention). A total 
of 19 LAs were excluded. Any school expressing an interest from one of these 19 LAs was allowed to 
take part provided they met all the other criteria in the sampling frame and understood that events (if 
applicable) would be in London and a location in the north of England, for example Leeds (although in 
the event, York was the northern location). In addition, schools were excluded that were already 
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taking part in research engagement programmes run by the EEF or the providers involved. These 
included: 


e schools involved in two evaluations already being run by NatCen as part of the EEF’s 
Research Use programme, namely the Inspirational Professional Learning Community 
Network (IPLCN) and the Ashford Teaching Alliance (ATA) Research Champion projects; 


e schools known to be receiving the IEE’s evidence-based emails and materials and also 
known to the EEF as being involved in research-engagement projects; 


e those involved in other current EEF Research Use projects; 
e those involved in the Institute of Education’s Research Learning Communities project; and 


e schools that were part of the National College’s Evidence Based Teaching projects’ Teaching 
Schools Alliances. 


Schools were recruited by NFER’s Research Operations Department in the autumn term of 2014. An 
initial sample of 7,884 schools was approached followed by a top-up sample of 2,922 schools. 
Samples were stratified by percentage FSM band and Key Stage 2 ability band. Local authorities 
were informed which schools in their authorities were being approached. All schools were approached 
with an initial invitation letter to the headteacher. NFER’s telephone unit then telephoned schools to 
engage them in the trial and support them to complete baseline data. In order to join the trial, and 
prior to randomisation, schools needed to complete the consent and reply form (outlined in the ethics 
section and included in Appendix C); this provided consent for NFER to access anonymized NPD 
data for Year 6 pupils in the school. Prior to randomisation, schools also needed to complete, if 
possible, at least two teacher baseline questionnaires (this was planned as five, but was reduced to 
help recruitment at scale—in the event some schools did not return baseline surveys but still joined 
the trial as they had completed the reply form). The questionnaires were targeted at senior leaders, 
literacy co-ordinators, and Key Stage 2 teachers. Schools were asked to nominate a Literacy Octopus 
lead as the main contact for the trial. 


The geographical location of schools was monitored during recruitment to ensure that targets were 
met for each region. Each of the nine arms of the trial required 60 recruited schools (30 from the north 
of England and 30 from the south) and the control group required 240 schools—a planned total of 780 
schools. Once this target had been met, and indeed exceeded (823 schools were recruited), schools 
were randomly assigned to one of the nine intervention arms of the trial or to the control group (which 
was expanded in size to 283 schools). After randomisation, the providers were each given a list of 
their allocated schools, and where teachers consented on the baseline survey to provide their name 
and contact details for the purposes of the trial, these were also passed to the relevant provider. Each 
provider then proceeded to implement their communication methods (outlined in the Intervention 
section earlier in this report). 


Outcomes measures 


Primary outcome 


The primary outcome was 2016 Key Stage 2 attainment in English, using administrative data 
available on the NPD, to obtain the sum of READSCORE (scaled score in reading) and GPSSCORE 
(scaled score in grammar, punctuation, and spelling). Writing was not included as this was teacher 
assessed and is therefore vulnerable to bias.2 Key Stage 1 attainment in literacy was used as a 
baseline. The development of pupil literacy was selected as the primary outcome in order to 
determine the impact of different approaches to communicating research evidence on attainment 
(note that all of the Literacy Octopus interventions related to evidence-based support for KS2 literacy). 


8 We considered using raw test scores as opposed to scaled scores since some information is lost in the scaling process. 
However, raw scores in the December 2016 version of NPD tables were not available: https://www.gov.uk/guidance/scaled- 
scores-at-key-stage-2. 
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Secondary outcomes 


In addition to the KS2 results of the 2016 cohort, the KS2 results from 2015 and 2017 will also be 
analysed and reported in the 2018 addendum report. As 2015 was the last year of the old KS2 tests, 
this outcome will therefore be KS2_ENGTOTMRK: total marks achieved in English test (sum of 
reading and writing tests). For 2017 data, the outcome measure will be the same as for 2016 and will 
be reported in an addendum report in 2018. We will measure the effect that different approaches to 
disseminating research evidence has on attainment over three Year 6 cohorts. 


Other secondary outcomes 


Other secondary outcomes that the trial has studied are degree of research use—in terms of 
awareness, understanding, and action. The Theories of Change for the Literacy Octopus interventions 
refer to these three areas of change at both individual teacher and school-wide levels. As all of the 
interventions aimed to involve teachers engaging in evidence-based materials, it was important to 
study Research Use outcomes as part of this trial. These have been measured by calculating the 
factor scores based on the teacher responses to the standard validated Research Use Outcomes 
Survey (Poet ef al., 2015, unpublished) administered in spring term 2016. Each teacher’s score for a 
given factor is formed by summing together the responses of the survey items that have loaded onto 
that factor. The following six factors are measured by the Research Use Outcomes Survey: 


e Measure 1: positive disposition to academic research in informing teaching practice; 

e Measure 2: uses academic research to inform selection of teaching approaches; 

e Measure 3: perception that academic research is not useful in learning; 

e Measure 4: perception that own school does not encourage use of academic research; 
e Measure 5: active engagement with online evidence platforms; and 

e Measure 6: your knowledge about research. 


The Research Use Outcomes Survey was in development while the trial protocol was being written 
(when the six measures were not yet fully specified), so, at the time of writing the protocol, we 
anticipated that the secondary outcome questions could cover the effectiveness of the different 
communication approaches in terms of schools’ dispositions towards research and the degree to 
which research information is informing teaching and learning. Measures 1 and 2 clearly met our 
suggested outcome questions and were therefore specified in the statistical analysis plan as the main 
Research Use outcomes for this trial. Measures 3-6 were explored as additional secondary 
outcomes. (See ‘Secondary outcomes analysis—research use’ section below for more detail.) 


Sample size 


The power calculations performed in the protocol indicated that with nine arms, each having 60 
schools, and a control group of 240 schools, the estimated minimum detectable effect size (MDES) of 
this trial would be 0.121. Assumptions underpinning this calculation were: an average cohort size of 
36 pupils per school, ICC = 0.15 (reduced from 0.2 through the use of KS1 attainment data as a 
covariate), correlation of 0.7 between KS1 performance and KS2 performance, power = 80%, and 
significance = 5%. This trial was originally designed to detect effect sizes of 0.2 in the active arms but 
this was revised down due to likely treatment dilution—when schools either send limited staff or no 
staff to attend the conferences. Furthermore, we introduced a large control group which reduces the 
MDES from 0.153 (for a control group of 60) to 0.121 (for a control group of 240). Data attrition was 
not anticipated to be a problem in this trial as the primary outcome data would be anonymized NPD 
data and therefore did not rely on schools providing pupil data. 
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Randomisation 


In this trial, 823 schools were randomly allocated to either one of the trial’s nine arms (60 schools per 
arm) or the control group (283 schools) via school-level stratified randomisation. The stratum used 
was geographical area, a binary variable indicating north or south location. School-level 
randomisation was used due to many of the communication strategies being evaluated involving 
whole-school ‘mobilisation’. As described in the section on Participant Selection, NFER’s Research 
Operations Department recruited schools to the trial. An NFER statistician undertook the 
randomisation to ensure independent allocation. Appendix E details the randomisation syntax used. 


Analysis 


The analysis followed EEF guidelines’ and the published SAP for this trial. This section provides an 
overview of the analysis undertaken. The SAP provides further detail (see footnote 7 above). 


Primary intention-to-treat (ITT) analysis 


The primary outcome analysis was ‘intention to treat’ (that is, the analysis of KS2 literacy attainment 
data from all schools that were randomly allocated to the intervention and control arms in the trial, 
regardless of level of compliance). A multilevel model with two levels (school and pupil) was used for 
the analysis to account for the cluster randomisation. The first step was to determine if different 
methods of communicating research had any significant effect on KS2 attainment. This was 
ascertained by fitting two models to the data: one which had KS2 attainment as the dependent 
variable and KS1 attainment as the independent variable—in addition to a dummy variable identifying 
if the school is in the north to account for the stratified randomisation; and a second model identical to 
the first but including nine dummy variables that indicated which group an individual is in (the default 
being control). A likelihood ratio test (LRT) between these models provided a global test for the impact 
of the communication strategies used within this trial. 


Analysis continued in terms of presenting effect sizes and confidence intervals for each arm versus 
control, regardless of the outcome of the global test. No post hoc tests were made between the trial’s 
intervention arms as the emphasis of the study was to determine the effectiveness of each individual 
approach when compared to the ‘business as usual’ control group. 


When calculating confidence intervals, the two-sided form of Dunnett’s Test (1955) was used to 
control the family-wise error rate. This test is specifically designed to handle the scenario of multiple 
comparisons being made to a common control group and is an exact test (that is, its family-wise error 
rate is exactly equal to alpha) and can be used in balanced and unbalanced designs. All multilevel 
analyses were carried using the R package nlme. Dunnett’s Test was used to control the family-wise 
error rate using the R package multcomp. 


Imbalance at baseline 


As the primary outcome was available from administrative data, it was anticipated that the level of 
missing data would not exceed 5% at either the school or pupil level. However, this was the case at 
pupil level (6%) so baseline imbalance was explored for ‘percentage everFSM’ and mean KS1 
attainment (see ‘Missing data’). 


Missing data 


As this analysis used administrative data, it was anticipated that the number of pupils missing would 
be very small and so these cases could be excluded from the analysis without risk of bias. It was 


1https://educationendowmentfoundation.org.uk/public/files/Evaluation/Writing_a_Research_Report/2015_Analysis_f 


or_EEF_evaluations.pdf 
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anticipated that the level of missing data would not exceed 5% at either the school or pupil level so no 
missing data analysis was planned. 


In the event, missing data occurred at a school level for reasons relating to matching during the NPD 
request and data release process.'! These included school closures and mergers and accounted for 
less than 1% (four schools) of the original 823. This was a very small fraction of a large number of 
schools and occurred for reasons very unlikely to be due to bias, entirely related to matching and 
administrative problems. Pupil-level missing data, however, through individuals not having a full 
complement of test scores at baseline and follow-up, accounted for 6%. We therefore explored the 
means of the analysed groups using school-level ANOVAs on percentage everFSM and mean KS1 to 
see if differences were due to chance or for some potentially biased reason. We do not believe data 
was missing for any biased reasons—Literacy Octopus communication strategies did not interfere 
with a school’s ability to administer KS2 tests in the usual way. 


CACE analysis 


Fidelity analysis was conducted on individual models containing data from each arm, in turn, with 
control schools—nine models. To do this, a variable that measured each school’s level of 
engagement with trial activity was matched with the attainment data. This variable measured four 
levels of engagement: none, low, mid, and high. While the quantitative and qualitative characteristics 
of each level of engagement were specific to the trial arm, we adopted an overall consistent approach 
to categorisation, namely: 


Level of engagement DY-Y-Yo al ol (Cola Mey m=lalef-Le(-lunl-val@-Cerelaelialemcomuileyalicolalareminiielarir-licolam lip metic] 
None No engagement 
Low Engaged less than expected for that arm 
Mid Engaged as expected for that arm 
High Engaged more than expected for that arm 


Each provider adopted a pre-agreed MI data tool, with pre-specified fields, in order to capture the 
qualitative and quantitative data required. Providers submitted this data at three time-points (April 
2015, July 2015, and April 2016). NFER researchers carried out the categorisation. Appendix F 
provides the detailed engagement schemes applied to each arm. 


In order to obtain a more accurate measure of the ‘pure’ dosage effect of each communication 
method on attainment, the CACE impact estimate was calculated. Because schools may potentially 
have unobserved characteristics that have an influence on both compliance with the trial and 
academic attainment, a two-stage least squares model was used to calculate the CACE estimate 
(Angrist and Imbens, 1995). 


Secondary outcome analyses—attainment 


Interim and follow-up outcomes using the KS2 English attainment data for the 2014/2015 and 
2016/2017 cohorts will be analysed and reported in the 2018 addendum report. The same analysis as 
that outlined in the primary intention-to-treat analysis for the 2015/2016 cohort will be applied. 


Secondary outcome analyses—research use 


All models using research use data were affected by the substantial levels of attrition experienced at 
follow-up survey administration. Each model was therefore proceeded by a multilevel logistic model 
consisting of two levels (school and teacher)—relating to whether or not an individual was missing at 


"1 Both the NPD team and NFER statisticians addressed the issue of URN (school Unique Reference Number) 
changes at academisation through the use of link files supplied by DfE. The missing schools are as a result of 
very recent academisation that ‘fall through the gap’ between NPD records and Edubase records. 
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follow-up—regressed on the covariates of the main model. This helped to determine the extent of 
bias. As some teachers at follow-up completed the survey who did not complete one at baseline, 
multiple imputation was used.'* The completers model described in the next paragraph was 
compared with one following imputation of the baseline measure. 


Each of Measures 1 and 2 from the Research Use Survey was the dependent variable in a separate 
multilevel model containing two levels (teacher and school). As we used this model only to estimate 
coefficients and their standard error, the variable cluster size apparent during the teacher survey 
should not be a problem, assuming missing at random (Snijders, 2005). The covariates that were 
entered into each model were the equivalent baseline research use measure and a dummy variable 
to identify schools in the north, as per the stratified randomisation. Analysis proceeded as for the 
primary outcome. 


The remaining Measures 3-6 were analysed at the school-level using ANOVA on follow-up data only. 


Path analysis 


Path analysis is planned as an attempt to uncover any link between the extent of research use and 
attainment via a multilevel model containing two levels (school and teacher). This will be conducted 
for the 2018 report. Each of the six measures of research use at follow-up will be entered into the 
model as teacher-level covariates along with the school mean for KS1 baseline. The dependent 
variable will be the school mean of the primary outcome variable. The model will be checked for 
tolerance as the research use measures are likely to be correlated. 


Subgroup analyses 


In accordance with the protocol for this trial and the most recent EEF analysis guidelines, subgroup 
analysis on the primary outcome was carried out for those pupils who had ever received free school 
meals—‘everFSM’.'3 This was done using a model identical to the primary outcome model but 
including everFSM and everFSM*intervention as covariates too. Analysis proceeded as per the 
original primary outcome modelling, that is, the first model was identical to the primary outcome model 
but with everFSM as a covariate. The second model contained a further 18 covariates: nine dummies 
for randomised group and nine interaction terms. A likelihood ratio test (LRT) was performed between 
the two models. 


Following EEF guidelines, a second model was also run including only everFSM children. Analysis 
proceeded as per the original primary outcome model. 


All effect sizes were calculated using total variance from a multilevel model, without covariates, as the 
denominator—equivalent to Hedges’ g. Confidence intervals for each effect size were derived from 
Dunnett’s Test (see above) that takes into account multiple comparisons. Dunnett’s Test applied to 
the model coefficients themselves. These were converted to effect-size confidence intervals using the 
same formula as the effect size itself. 


12 The SAP specified multilevel multiple imputation. The imputation was run on a multilevel model but the 
imputation model itself was single level as it was not possible to converge the multilevel model. MLwiN macros 
available from missingdata.org were used for this purpose. 

13 We used the NPD EVERFSM_ALL variable which entails a flag to indicate if a pupil has ever been recorded as 
eligible for free school meals on the census day in any spring census up to the pupil's current year (not including 
nursery). 


Education Endowment Foundation 19 


Evidence-based Literacy Support — the ‘Literacy Octopus’ Trial 


Implementation and process evaluation 


Overall Implementation and Process Evaluation 


The implementation and process evaluation (IPE) for the Literacy Octopus'* involved a focus on 
understanding if and how teachers and schools engaged with the interventions, and crucially, what 
they did next, if anything, in terms of practice. We explored perceived impacts in terms of any 
changes in awareness, understanding, or action relating to KS2 literacy and research use, and we 
examined the mechanisms underpinning those changes. The IPE involved: 


1. facilitating two developer workshops: the first pre-protocol to discuss the detail of each 
intervention arm, to inform NFER’s recruitment strategy, and the second—during the recruitment 
phase—for providers and NFER to discuss next steps, communication strategies between NFER, 
providers and schools, and data sharing protocols; 

2. developing a Theory of Change for the trial and for each arm: to set out how the dissemination 
activities would be implemented and how they would support change; these were developed by 
NFER collaboratively with each provider through a telephone interview and email exchanges (see 
Appendix D); 

3. specifying an engagement data-collection tool for each provider, with pre-agreed engagement 
descriptors to signify ‘high’, ‘mid’ and ‘low’ engaged schools, and conducting independent 
analysis of the engagement data submitted by providers for each arm—to inform the on-treatment 
analysis (the engagement schemes are set out in Appendix F; NFER specified the detailed 
quantitative cut off points for each engagement level); 

4. reviewing cost data provided to NFER by schools and developers relating to each intervention 
arm—to understand the costs to schools in terms of resources and time, in the trial and as would 
be in the real world, and calculating a cost per pupil per arm (See Methods: Costs); 

5. observing events (one event in each of the active arms) including post-observation interviews with 
event deliverers and carrying out a review of intervention materials in order to understand delivery 
and inform fidelity; 

6. carrying out telephone interviews with schools (approximately three teachers per arm; 29 
conducted in total) to explore engagement and fidelity, views on implementation, dissemination 
method, and content, and initial perceived outcomes on teaching, learning, and schools; 

7. carrying out telephone interviews with providers in summer 2015 (four interviews) and again in 
summer 2016 (four interviews) to explore implementation, fidelity, perceived outcomes, 
sustainability, and scalability; 

8. carrying out case-study visits and light touch interviews with a sub-sample of schools (ten, 
including six in-depth and one light-touch case-study with schools engaged in the active arms, 
and one in-depth and two light-touch case-studies with schools engaged in the passive arms) to 
understand in more detail implementation, barriers, conditions for success, perceived outcomes, 
and the perceived mechanisms for change (we conducted 28 interviews in total—see Appendix H 
for details of the process interview samples achieved); from these we produced examples of 
detailed impact trails—one per provider exploring mechanisms for change (see Appendix |); 

9. interviewing a small number of control-group schools (six) to explore KS2 literacy improvements 
and research engagement with the control group; and 

10. including, as an addition to the protocol, some IPE questions in the Research Use Outcomes 
Survey—for both intervention and control group teachers—to ascertain extent of engagement, 
views on the materials and dissemination approaches, perceived outcomes, and formative 
recommendations (see Perceived Outcomes section). 


14 The IPE for the Literacy Octopus was designed and implemented before the EEF’s IPE guidance was 
published. Hence we have not attempted to map the design to the guidance. We have, however, highlighted how 
we approached issues of fidelity, compliance, heterogeneity, engagement, and perceived outcomes. 
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All observations, telephone interviews, and case-study visits were conducted by NFER researchers 
(including the Trial Manager and members of the Literacy Octopus IPE team). Further details of the 
above methods are outlined in appendices as indicated. 


Implementation and Process Evaluation sampling approach 


Telephone interviews 
Schools for telephone interview were sampled to provide: 
e three schools per arm; 


e one school from each category of high, medium, and low engagement with the Literacy 
Octopus arm they were assigned to (see Appendix F for categorisation schemes); where 
there were no or very few cases of a particular engagement category from which to sample, 
the closest alternative category was chosen (for example, where there were few schools with 
‘high’ engagement, we selected ‘medium’ engagement); 


e aspread of schools in the north and south of England; 
e aspread of schools with varying proportions of FSM pupils; and 
e aspread of schools with varying numbers of pupils (School size). 


An initial sample was drawn to meet these criteria and subsequently a further top-up sample was 
identified to boost recruitment. In addition, to boost recruitment, schools were invited to feedback their 
views via either a telephone interview or a short online survey. We approached the named Literacy 
Octopus lead member of staff in each school and invited them to feedback about their experiences of 
the intervention they were assigned to. We achieved a sample of 29 respondents (18 telephone 
interviews and 11 responses to the same questions via Survey Monkey) and largely met the above 
specification wherever possible. 


Case-study schools 


Case-study schools were sampled to enable us to investigate examples of a Literacy Octopus 
intervention leading to impacts on practice. These were selected on the basis of: 


e ‘high’ (as defined above) levels of engagement with the Literacy Octopus intervention they 
were assigned to (although we did approach some mid-engaged schools where we had 
exhausted the list of high engagement); and 


e endpoint survey responses indicating that there had been changes to practice, including: 


© respondent indicating that they had taken part in the Literacy Octopus Arm the school 
was allocated to; 


o reporting some perceived impacts (for example, to discuss best practice, share 
learning, reflect on practice, change classroom practice, reinforce existing practice, 
conduct own research, influence colleagues’ practice, improve knowledge of KS2 
literacy); 


o indicating that they had sought further evidence, research, or information about KS2 
literacy or other topics; or 


o that they would recommend the intervention to other schools, and their reasoning. 


In addition, we took into account telephone interview responses indicating that there had been 
changes to practice (it was not a prerequisite that the schools had taken part in the previous round of 
telephone interviews). In our selection we also took into account providers’ information about schools 
that they knew had made changes to practice (although we avoided a number of providers’ case- 
study schools so as to minimise burdening schools with evaluation requests). 
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We selected a mixture of schools based in the south and north of England. We initially approached 
the Literacy Octopus named co-ordinator in each school to invite them to tell us more about what they 
had implemented as a result of engaging with the intervention, and how the changes had come about. 
To boost recruitment, we added to the initial sample, sometimes selecting schools where there was 
less overt evidence of impact (though still evidence of engagement with the programme). We aimed 
to achieve a sample of 14 case studies in total (ten in active arms; four in passive arms). We 
conducted a total of ten case studies (seven active; three passive) which was fewer than planned due 
to challenges in identifying and recruiting sufficient schools that had both engaged to a high level in 
the interventions and that had indicated through telephone and survey responses some changes in 
practice. Although we approached mid-engaged schools where the list of high-engaged schools was 
exhausted, we experienced challenges to recruitment, including no response or schools declining to 
take part as they did not feel there had been sufficient impact to participate as a case study. Where 
possible, we conducted interviews with the literacy co-ordinator, headteacher (or other senior leader), 
and a class teacher who had used the materials or implemented changes as a result of the 
intervention. Where possible, we also observed evidence of the change to practice (such as teaching 
resources, pupils’ work, and planning documents). 


Control group schools were selected for telephone interview based on their responses to the endpoint 
survey to enable us to find out more about how schools engaged with evidence-based strategies. We 
selected control schools that appeared to be research-engaged based on their endpoint survey 
responses, that they either: 


e identified academic research, online evidence platforms, and CPD-based on research as 
sources of evidence; or 


e had engaged with support for KS2 literacy in the previous 12 months which they rated as 
‘quite’ or ‘very’ good; or 


e recommended the use of research evidence, or similar, to other schools as a way of 
supporting improvements in KS2 literacy. 


We selected a mixture of schools based in the south and north of England. Where more than one 
member of staff from a school responded to the survey, we selected the most senior respondent to 
approach regarding a follow-up telephone interview. To boost recruitment, we drew a top-up sample 
which included schools demonstrating less engagement with research-evidence based on their 
survey responses. We aimed to conduct five telephone interviews with control schools; we achieved 
six. 


Costs 


Information on the costs to schools of taking part in the interventions was collected via a standard 
proforma as well as during interviews in the ten case-study schools, supported by discussion with an 
NFER researcher. This information covered all four Literacy Octopus providers, however we were not 
able to collect this cost information directly for Arm 2 (IEE Evidence Fair) and Arm 5 (CEM booklet) as 
we were unable to recruit case-study participants for these arms. In these cases we have provided a 
cost estimate based on supplementary information from schools regarding other arms and similar 
activities, as well as from the providers. (The original intention to collect this data via the outcomes 
survey was not possible as the Research Use Survey used was a standard instrument with limited 
space for additional process questions other than those already specified.) We also asked the 
Literacy Octopus providers for information about costs to schools in the real world (for example, 
subscription fees and conference fees) to help estimate the continuing costs to schools of engaging in 
Literacy Octopus activity in subsequent academic years. (The original intention to collect delivery 
costs borne by the provider is not a requirement of the subsequently published EEF cost guidance, 
and was not gathered or reported.) As the information on the costs to schools of taking part in the 
interventions is based on a small number of cases, it provides indicative information only. 
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The standard proforma for case-study schools collected information for two time periods: (1) the 
intervention delivery period (spring and summer term 2015), and (2) follow-up activity (autumn term 
2015 to summer term 2016). The questions included in the proforma are presented in the box below. 


(1) What resources did you need to enable your school to engage with the materials/activities as part 
of the Literacy Octopus? (e.g. computer hardware/software, technology, printing, 
paper/card/materials, etc)? 


2 (a) Please tick if these resources were additional to any resources you already have. 


(b) If resources were additional, what costs and expenses did you incur financially over the course 
of the trial for these resources? (e.g. cost of new software to enable participation, fees for 
services; any travel expenses; any staff cover costs; etc). 


(3) How much time did your school invest in engaging with the materials/activities as part of the 
Literacy Octopus? (e.g. reading materials, attending events, planning and preparation, running 
and attending meetings/training, CPD, lesson planning, etc) (for all staff involved, not just you) 

4( a) Please tick if this time was over and above any time you would normally spend on such activities 

(e.g. additional preparation time, additional time for which cover had to be arranged) 

(b) If time was additional, how much additional time did you/your staff contribute (and/or how much 

cover time did you have to arrange)? 


The questions asked of delivery partners about costs to schools are set out in the box below. 


DELIVERY PARTNER INTERVIEW SCHEDULE (SUMMER 2016) 
D. COSTS TO SCHOOLS 


1. As part of the evaluation for EEF, we need to report how much it would cost a school to be 
involved in your activities, in the real world. Could you summarise any costs to schools for each of 
your arms? 


1.1. Direct costs to schools, e.g. fees/subscription/buying-in training/purchasing 
resources/travel/subsistence/supply cover/other? 


1.2. Extra staff time, e.g. any additional staff time schools have had to spend on Octopus activity 
this year? 


Probes: are these essential to Octopus activity? or optional costs for schools? 


As the Literacy Octopus protocol was published prior to the EEF guidance on cost evaluations, no 
provisions were made in the protocol to calculate a cost per pupil of a school taking part in each type 
of intervention. In addition, as all of the interventions were anticipated to be ‘light touch’ and 
inexpensive (in terms of monetary cost), reporting descriptive information on costs at a school level 
(rather than costs per pupil) was felt to be more appropriate. However, in line with EEF guidance, we 
attempt to calculate a cost per pupil in each of the intervention arms. This involved three steps: 


e Collate information on resource and monetary costs to a school from case-study schools for 
taking part in the ‘intervention’ (assume engagement as expected, i.e. mid levels of 
engagement) (i.e. spring/summer term 2015) — create an average estimate = Arm A, year 1 
estimate (provide qualitative description of time input at expected levels of engagement) 


e Collate information on resource and monetary costs to a school from case-study schools for 
ongoing engagement/follow-up implementation activity (i.e. autumn term 2015 — summer term 
2016) — create an average estimate = Arm A, year 2 estimate (provide qualitative description 
of time input at expected levels of engagement) 
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e Estimate ongoing activity (e.g. annual subscription fees etc) — Arm A, year 3 estimate. 


The interventions are KS2 literacy interventions. For the sake of simplicity, we have assumed that the 
interventions are KS2-wide, rather than focused on specific year groups, classes’ or individuals. 
Based on the school and pupil numbers in our study, we have calculated an average number of pupils 
in KS2 primary schools to be 160 (four year groups with on average 40 pupils per year group).'° 


In addition, for each arm, we have provided a description of the actual costs to intervention schools 
during the trial, and the time that teachers and schools put into engaging with Literacy Octopus. 


15 Note, there was data for 32,613 Year 6 pupils from the 819 identifiable Literacy Octopus schools (according to 
NPD data used); on average, 40 pupils in each Year 6 group. 
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Timeline 
Table 2: Timeline 
Activity 


Summer term 2014 Write, agree, and register protocol. 
Devise sample and recruitment strategy. 
Develop ToC(s) with providers; evaluation workshop with providers. 


Autumn term 2014 Recruit schools; MoUs; teachers complete Research Use baseline survey. 
pene Randonlly allocate schools to 1 of 10 groups (Dec 2014). 
Spring term 2015 Providers start interventions and host events. 
Observe events in active arms (N = 4); post-observation interviews with 
providers (N = 4). 
Collect interim monitoring data from providers. 
Summer term 2015 Telephone interviews with sub-sample of participants (N = 29). 
Collect end of evaluation Year 1 monitoring data. 
Interviews with providers (N = 4). 


Autumn term 2015 Interim process analysis; engagement data coding. 


Spring term 2016 Teachers complete Research Use outcomes survey. 
Collect end of Year 2 monitoring and costs data from providers. 
Engagement data coding. 

Summer term 2016 Providers continue interventions. 


Follow-up interviews with providers (N = 4). 


School case studies (N = 10). 
Control group interviews (N = 6). 


Autumn term 2016 Providers continue passive elements of support to schools (e.g. sending 
regular materials) and respond to requests from schools. 


Process analysis. 
Write and agree SAP. 

Spring term 2017 Intention-to-treat analysis using NPD data (primary outcome 2015/2016). 
Survey analysis (Secondary outcome). 
CACE analysis (primary outcome/engagement data on-treatment analysis). 
Sub-group analysis (everFSM). 
Write report. 


Autumn term 2017 Publish main report. 
Further activities to be undertaken (2017-2018) 


Autumn term 2017 Feedback to schools. 


Spring term 2018 Intention-to-treat analysis using NPD data (interim 2014/2015 and follow- 
up 2016/2017). 


Path analysis. 
Write report. 


Summer term 2018 Publish addendum report. 


NFER timetable for the Literacy Octopus. Evaluation activity shown in normal font. Provider/intervention activity shown in italics. 
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Impact evaluation 


Participants 


Figure 1 provides details of the participant flow through each stage of the Evidence-based Literacy 
Support — the ‘Literacy Octopus’ Trial. High levels of school and pupil-level follow-up were guaranteed 
for the primary outcome through the use of NPD data. This is described in the section on pupil 
characteristics. 


In terms of the secondary Research Use outcomes, the follow-up survey response rate was lower 
than expected. NFER put in place additional reminding strategies, sent paper versions of the surveys 
to teachers and schools who had not yet completed the follow-up survey, and extended the time 
period that the survey was open. A total of 2,041 teachers from 688 schools completed baseline 
surveys and 557 teachers from 386 schools completed endpoint surveys. Attrition in terms of school- 
completion of surveys between baseline and endpoint was higher than anticipated at 44% (we 
anticipated 30% attrition). A total of 335 teachers completed surveys at both time-points. 


Table 3 provides details of minimum detectable effect sizes at different stages in the trial. 
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Figure 1: Participant flow diagram 
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Did not agree to participate 
(school n=334) 
No response (school n=9386) 


Agreed to participate 
(school n=910) 


Did not meet inclusion criteria® 
(school n=87) 

Met inclusion criteria (school 
n=823) 


Schools put forward for 
randomisation (school 
n=823) 


4 The total sample was constructed of a main sample drawn initially and a further top-up sample to increase recruitment of schools to the trial. 
> The criteria for including schools in the trial and putting them forward for randomisation was the completion of a Memorandum of Understanding (MoU) reply form giving the 
Headteacher’s consent to participate. 
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Randomised° 
(school n=823) 


Arm1 Arm 2 Arm 3 Arm 4 Arm 5 Arm 6 Arm 7 Arm 8 Arm 9 Arm 10 
IEE IEE active How2s How2s CEM CEM CEM ResearchEd ResearchEd Control 
passive (school passive active passive active active passive active (school 
(school n=60; bl (school (school (school (school (school (school (school n=283: bl 
n=60; bl survey n=60; bl n=60; bl n=60; bl n=60; bl n=60; bl n=60; bl n=60; bl survey 
survey school survey survey survey survey survey survey survey school 
school n=55; school school school school school school n=41; school n=50; n=237; 
n=51; teacher n=51; n=51; n=48; n=55; n=49; teacher teacher teacher 
teacher teacher teacher teacher teacher teacher 


(school (school (school (school (school (school (school (school (school Heal 

n=60) n=60) n=59)! n=60) n=60) n=60) n=60) n=60) n=60) ae i 
(pupil (pupil (pupil (pupil (pupil (pupil (pupil (pupil (pupil nS Pup! 
n=2384)° n=10917) 


Not 
analysed9 analysed analysed analysed analysed analysed analysed analysed analysed analysed 
n=93 n=170 n=137 n=122 n=123 n=151 n=129 n=158 n=98 n=637 


Arm1 Arm 2 Arm 3 Arm 4 Arm 5 Arm 6 Arm 7 Arm 8 Arm 9 arm40 
Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: 
NPD NPD NPD NPD NPD NPD NPD NPD NPD NPD 
n=2291 n=2203 n=2337 n=2448 n=2174 n=2080 n=2386 n=2474 n=2122 n=10280 


Analysed: 


° This sample comprised 402 schools in the North and 421 schools in the South. Schools were randomised to the intervention arms within their North and South group to give 
an equal, or almost equal, spread of each location per arm. 

4 Sixty schools were randomly allocated to each intervention arm. Some schools that returned an MoU reply form did not subsequently return any baseline teacher surveys so 
the number of schools returning at least one baseline teacher survey is lower than the number allocated in the randomisation process. 

® The number of pupils with KS2 data on NPD list provided by DfE. 

fOne school in Arm 3, and three schools in the control could not be found on the NPD list provided by DfE, possibly due to changes in school name (for example due to 
mergers, academisation, or closure). 

9 Not analysed as pupils did not have a full complement of test score (that is, missing attainment data on the NPD at either KS1 or KS2). 
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Withdrew" Withdrew Withdrew Withdrew Withdrew Withdrew Withdrew Withdrew Withdrew Withdrew 
n=3 n=0 n=0 n=1 n=9 n=6 n=10 n=1 n=2 n=9 


Arm1 Arm 2 Arm 3 Arm 4 Arm 5 Arm 6 Arm 7 Arm 8 Arm 9 
(ep survey (ep survey (ep survey (ep survey (ep survey (ep survey (ep survey (ep survey (ep survey Arm 10 
school school school school school school school school n=14; school n=20; (ep survey 
n=26; n=22; n=23; n=23; n=22; n=24; n=21; teacher teacher school n=191;: 
teacher teacher teacher teacher teacher teacher teacher teacher 


Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: Analysed: 
Matched Matched Matched Matched Matched Matched Matched Matched Matched Analysed: 
surveys surveys surveys surveys surveys surveys surveys surveys surveys Matched 
teacher’ teacher n=20 || teacher n=23 || teacher n=16 |} teacher n=14 || teacher n=21 || Teacher n=20 |] teacher n=14 teacher n=13 surveys 


n=22 teacher n=172 


‘This is the number of schools that actively withdrew during the course of the trial. Where given, reasons for withdrawing included staff changes and being too busy with other 
priorities (such as Ofsted inspection). A further proportion of schools/teachers did not respond at follow-up and did not complete the endpoint survey. 

' Analysis of survey outcome Measures 1 and 2 was conducted on matched baseline and endpoint teacher surveys (N shown in section ‘teacher outcome survey analysis’). 
Analysis of survey outcome Measures 3 to 6 was conducted on total teacher endpoint (ep) survey responses (N shown in section ‘teacher outcomes survey’). 
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Table 3: Minimum detectable effect size at different stages 


n schools Correlation =I Cored (ale 
(n=eacharm) between Sieliicer-lice a 
n pupils pre-test & or pair 
(n=eacharm) post-test ureicoal are) 

780 (60 per 


intervention; 240 Stratified by 
Protocol control) 0.70 : region 


(north/south) 
28,080 


Power Alpha 


823 (60 per 
intervention; 283 


control) Stratified by 


Randomisation 32,613 : : region 
(minimum (north/south) 
2,220; 10,917 
control) 


819 (60 per 
intervention 


aside from one - 
with 59: 280 Stratified by 


: . region 
sae ata (north/south) 


2,080; 10,280 
control) 


Analysis (i.e. 
available pre- 
and post-test) 


Source: NFER Literacy Octopus Evidence-based Literacy Support Trial, 2016. 
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Pupil characteristics 


Randomised schools that could not be matched to the NPD for reasons such as closures and 
mergers accounted for less than 0.5% (four schools) of the original 823—a very small fraction of a 
large number of schools. We used Edubase to try to identify these schools and explore possible 
reasons for being missing at school level. Reasons were entirely down to changes in school 
organisation, such as closures, mergers, or academisation. There were no biased reasons. We have 
therefore omitted school-level factors from the baseline comparison table. Pupil-level missing data, 
however—through individuals not having a full complement of test scores at baseline and follow-up— 
accounted for 6% (1,818 of the 32,613 pupil results in the trial). We have therefore included two 
comparisons of school means on analysed groups in Table 4. To interpret this table, it is helpful to 
know that the overall standard deviation of the percentage everFSM and KS1 school mean measures 
was 20.7 and 2.53, respectively. 


An ANOVA on percentage everFSM (df = 9, F = 0.991, p = 0.446) suggested that the differences we 
see were due to chance. The ANOVA on KS1 school means (df = 9, F = 1.321, p = 0.222) also 
suggested that any differences were due to chance. In terms of a pre-test effect size on analysed 
groups, the largest was 0.370, between the ‘ResearchEd conference and online community’ and 
control. While this seems large, it is probably due to chance imbalance at baseline exacerbated by the 
fact that there were nine groups to be compared to control. 


Table 4: Baseline comparison 


How2s 
Pupil-level 10] 0) ole) alae | 
variable IEE materials IEE evidence fair | How2s website access website CEM booklet (Arm 
(analysed (Arm 1) CNW) (Arm 3) Nu )) i) 
F\ sel nlofe) bo n n n n n 
ENZ=1)) (missing) mean (missing) mean (missing) mean _ (missing) mean (missing) | mean 
Percentage 
SCuIEERI 60 (0) 26.3 60 (0) 31.7 59 (1) 26.0 60 (0) 31.9 60 (0) 26.9 
ae a 60(0) | 31.39 60(0) 3080 59(1) 31.76 60(0) 31.13 60(0) | 31.18 
ResearchEd 
CEM booklet (07 SM efoto) a(-iar-Lale| ResearchEd conference and 
and light CPD F-YoME-Talex-ve Im @] od B) conference online community 
(Ug) (Arm 7) Udi) (Arm 9) Control 
n n n n n 
(missing) mean (missing) mean (missing) mean _ (missing) mean (missing) | mean 
Percentage 
eve pre 60 (0) 26.3 60 (0) 29.4 60 (0) 29.8 60 (0) 25.0 280 (3) 30.0 
KS1 school 
ieait 60 (0) 31.42 60 (0) 31.16 60 (0) 31.31 60 (0) 31.97 280 (3) 31.03 


Source: NFER Literacy Octopus Evidence-based Literacy Support Trial, 2016. 
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Outcomes and analysis 


Primary outcome and analysis—Key Stage 2 literacy attainment outcomes 
2015/2016 cohort 


The main result was from a likelihood ratio test (LRT) between a multi-level model containing just 
baseline and region and one containing baseline and region plus all the intervention groups as 
covariates. This yielded a likelihood ratio of 2.66 (df = 9, p = 0.98), meaning that we accept the null 
hypothesis and there is no evidence of any effect of the various interventions. In order to present 
effect sizes, confidence intervals were calculated using Dunnett’s Test to correct for multiple inference 
and the results are presented in Table 5a and Figure 2a. 


The interaction between intervention group and everFSM was explored in a similar way using a 
likelihood ratio test between a model containing just baseline, region, and everFSM and one with 
these variables plus all intervention groups and interaction terms. This yielded a likelihood ratio of 
11.1 (df = 18, p = 0.89) meaning that we accept the null hypothesis and there is no evidence of any 
interactions. In order to present effect sizes, models were run on only everFSM children. Confidence 
intervals were calculated using Dunnett's Test to correct for multiple inference and the results are 
presented in Table 5a and Figure 2b. 


Table 5a: Primary analysis—effect size versus control (Dunnett’s multi-level confidence 
intervals); numbers of schools (n) and pupils (N) 


IEE materials IEE evidence fair (Arm How2s website How2s support and CEM booklet 
(Arm 1) 72) access (Arm 3) website (Arm 4) NGii)) 
| ES | ES nN =) | =ts) nN ES 
Outcome (missing) (95% Cl) (missing) (95% Cl) (missing) (95%Cl) (missing) (95% Cl) (missing) (95% Cl) 
KS2 n 60 (0) -0.02 n 60 (0) -0.04 n 59 (1) 0.00 n 60 (0) -0.03 n 60 (0) 0.00 
Enalish N 2,291 | (-0.14, N 2203 (-0.15, N 2,337 (O12, N 2,448 (-0.14, | N2,174(123))  (-0.12, 
g (93) 0.10) (170) 0.08) (137) 0.11) (122) 0.09) 0.11) 
KS2 n57(3) | -0.03 -0.05 -0.07 0.01 0.02 
English N 526 : n 58 (2) : n 57 (3) : n 59 (1) : n 56 (4) : 
(everFSM (0.19, | N 747 (55)) (0.19, | N553(39)) (0-22, | N764(44) | (0.15, | N574(37) | (0.17, 
only) (28) 0.12) 0.10) 0.08) 0.14) 0.14) 
CEM booklet and (eS efoto) <(-1ar-Lale| ResearchEd ResearchEd 
light CPD E-Ye NVE-Laler=\e I @7 od B) conference conference and online 
(Arm 6) (Arm 7) (Arm 8) community (Arm 9) Control 
nN ES | ES | ES nN ES | 
Outcome (missing) (95%Cl) (missing) (95% Cl) (missing) (95%Cl) (missing) (95% Cl) (missing) 
KS2 n 60 (0) 0.03 n 60 (0) 0.03 n 60 (0) 0.00 n 60 (0) 0.01 n 280 (3) 
Enalish N 2,080 | (-0.09, N 2,386 (-0.09, N 2,474 (-0.11, N 2,122 (-0.11, N 10,280 
g (151) 0.14) (129) 0.14) (158) 0.12) (98) 0.13) (637) 
KS2 
English "Pega! Cai, ||. R590). | cana, || M88@) | ocoag, | mb8@ | ogg | meracis) 
acide (44) 0.20) N 668 (42) 0.19) N 794 (38) 0.18) N 537 (34) 0.18) N 3,094 (219) 


Source: NFER Literacy Octopus Active Trial, 2016. 
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Figure 2a: Primary analysis—effect size versus control (Dunnett’s multi-level confidence 
intervals) 
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Figure 2b: FSM-only analysis—effect size versus control (Dunnett’s multi-level confidence 
intervals) 
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We also present the raw unadjusted means using KS2 English attainment data for all pupils in trial 
schools in the NPD 2015/16 KS2 dataset (i.e. prior to KS1-KS2 match) — so that the reader can see 
the raw results prior to complex modelling. The results are shown in Table 5b. Missing data relates to 
the number of pupils missing KS2 data on the NPD list provided by DfE. 
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Table 5b: Raw means—unadjusted confidence intervals; numbers of pupils (N) 


IEE materials IEE evidence fair How2s website How2s support and CEM booklet 
(Arm 1) (Arm 2) access (Arm 3) website (Arm 4) (Nites) 
Ouicome N mean N mean N mean N mean N mean 
(missing) (95% Cl) (missing) (95%Cl) (missing) (95% Cl) (missing) (95% Cl) (missing)| (95% Cl) 

Ks2 207.3 204.5 207.0 205.7 206.4 
English 2,350 (34)| (206.6, | 2,314(59)) (203.8, | 2,421 (53) (206.3 2,519 (51)| (204.9, | 2,245 (52)) (205.7, 
9 208.1) 205.2) 207.7) 206.4) 207.4) 
ata 199.6 199.4 198.2 199.2 199.4 
(everFSM 536 (18) (197.9, 768 (34) (198.0, 566 (26) (196.4, | 781 (27) | (197.7, | 587 (24) | (197.8, 
only) 201.3) 200.8) 200.0) 200.7) 200.9) 


(e) =" efoto) a(-1ar-Lale| (ei efoto) <(-1ar-Lale| ResearchEd ResearchEd Control 
light CPD Ee ME-Laler-\e I @7 od B) conference conference and (CAL) 
online community 


Outcome mean (95% 
((rlesiale)) Cl) (missing) (95% Cl) (missing) (95% Cl) (missing) (95% Cl) (missing) (95% Cl) 

KS2 206.3 207.0 206.3 10.677 
English 2,173 (58)| (205.6, |2,470(45); (206.3, | 2,583 (49)) (205.6, 2,182 (38)) (206.5, (2. 40) (205.4, 

9 207.0) 207.7) 207.0) 208.0) 206.1) 
Se 200.6 200.7 202.0 201.9 3.186 199.9 
(everrsm 612 (20) (199.1, 689 (21) | (199.3, 811 (21) | (200.7, 549 (22) (200.2, (127) (199.2, 
aniy) 202.0) 202.2) 203.2) 203.6) 200.6) 


Source: NFER Literacy Octopus Evidence-based Literacy Support Trial, 2016 


Education Endowment Foundation 34 


Evidence-based Literacy Support — the ‘Literacy Octopus’ Trial 


Figure 3 summarises the results of the CACE analysis. Separate models were constructed for each 
comparison with control. These should be viewed in the context of the descriptive statistics 
concerning intervention delivery intensity in Figures 5 and 6 (see Process Section). The numbers of 
schools upon which the analysis is based are the same as those in Figures 5 and 6 aside from for the 
How2s Passive group which is based on 59 schools rather than 60 as one school was lost in this 
group during the match to NPD. The control group contained 280 schools. 


It is interesting to note the wider confidence interval found for the intervention involving the 
ResearchEd conference and webinar support. This arm in particular had relatively low and relatively 
distinct levels of engagement—schools either attended the conference or did not. While other active 
arms involved receiving information (like booklets, research emails, and posters) even where schools 
did not attend the active element of that intervention (for example, a CPD session or an evidence fair), 
the ResearchEd active arm did not involve any ongoing dissemination activity to schools. Hence, 
schools that did not attend the conference or webinar did not receive any other material. The wider 
confidence interval could be indicative of this more dichotomous intervention approach. 


Figure 3: CACE analysis—effect size versus control (multi-level confidence intervals) 


CACE estimate of effect size between control group 
and trial arm 


IEE Materials 4) 
IEE evidence fair (1) 
How2s website access (——#+—_) 
How2s support and website ( # ) 
CEM booklet + 
CEM booklet and light CPD (—+—_) 
CEM booklet and advanced CPD ps, 
ResearchEd conference (++ 
——— 
i _1T 1 1 7 9 


ResearchEd conference and online community 


0.35 -0.21 -0.07 0.07 0.21 0.35 


Effect size 
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Secondary outcomes—research use 


The trial also explored, as a secondary outcome, the impact of the Literacy Octopus interventions on 
teachers’ and schools’ degree of research use in terms of awareness, understanding, and action. 
Degree of research use was measured by creating the factors (or measures) based on teacher 
responses to the standard Research Use Outcomes Survey (Poet et al., 2015) incorporated into the 
teacher baseline and endpoint questionnaires. Each teacher’s score for a given factor is formed by 
summing together the responses of the survey items that have loaded onto that factor (or measure). 
We explored the following six measures of research use: 


e Measure 1: positive disposition to academic research in informing teaching practice; 

e Measure 2: uses academic research to inform selection of teaching approaches; 

e Measure 3: perception that academic research is not useful in learning; 

e Measure 4: perception that own school does not encourage use of academic research; 
e Measure 5: active engagement with online evidence platforms; and 

e Measure 6: your knowledge about research. 


A total of 2,041 teachers from 688 schools completed baseline surveys and 557 teachers from 386 
schools completed endpoint surveys. The achieved response rate was lower than expected (we 
aimed to achieve a baseline response from all signed-up schools) and attrition, in terms of school- 
completion of surveys between baseline and endpoint, was higher than anticipated at 44% (we 
anticipated 30% attrition). A total of 335 teachers completed surveys at both time-points. In 
accordance with the SAP, the main analysis of research use involved Measures 1 and 2 and was 
conducted using a multilevel model using the matched baseline and endpoint teacher responses (N = 
335). These measures most closely related to the outcome areas highlighted in the protocol and were 
hence the main secondary outcomes. Measures 3 to 6 were additional secondary outcome measures, 
and, as specified in the SAP, were analysed using ANOVA on the endpoint teacher survey responses 
only (N = 557). Constituent items, and the reliability of the measures used in the analyses, are 
presented in Appendix G. Note that the reliability of these measures, as indicated by Cronbach’s 
alpha, is in line with the reliability as found in the original Research Use Survey (Poet ef al, 2015, 
unpublished) and is moderate for Measures 2-6 (0.63, 0.55, 0.55, 0.69, and 0.62, respectively) and 
reasonably high for Measure 1 (0.83). 


We conducted multilevel modelling of Measures 1 and 2 to ascertain the impact of the intervention 
over control Arm on teachers’ research use. Each model contained two levels (school and teacher) to 
account for the clustering. Model covariates included the school location (North or South of England) 
to account for the stratified randomisation, pre-test measure (score for the given outcome measure at 
baseline), and nine dummy variables to indicate randomisation arm (control arm being the base 
case). Analysis was then carried out as for the primary outcome measure. 


Like the primary outcomes analysis, the main result was from a likelihood ratio test (LRT) which 
suggests that there is no evidence of an effect of being in the Literacy Octopus intervention arms 
(Likelihood Ratio (LR) = 5.21, df = 9, p = 0.82) for Measure 1 (‘positive disposition to academic 
research in informing teaching practice’). In order to present effect sizes, confidence intervals were 
generated using the two-sided Dunnett's Test for the multiple comparisons. For Measure 1, these 
results are presented in Table 6 and Figure 4a. 


The above model, the ‘completers model’, included teachers who responded to the surveys at both 
time-points (n = 335). Outcomes from this model were compared with a multiply-imputed model which 
includes al/ the teachers who responded at follow-up (n = 557). Hence, ‘the imputed model’ included 
222 respondents who had an outcome measure at follow-up but did not have a baseline measure. 
This comparison indicated that the effects of the interventions were attenuated in the imputed model. 
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For instance, when considering intervention Arm 9 (ResearchEd conference and online community), 
the raw coefficient reduced from 0.67 in the completers model to 0.21 in the imputed model. 


Similar models were run for outcome Measure 2 (‘use of academic research to inform selection of 
teaching approaches’). Results from the LRT indicated that the model with and without the 
intervention arms as covariates were significantly different—LR = 19.49, df = 9, p = 0.02—suggesting 
that there were some between-intervention-arm differences in using academic research to inform 
selection of teaching approaches. However, outcomes from the two-sided Dunnett’s Test indicated 
that the confidence intervals straddle zero and therefore there is no evidence of impact of any 
intervention, when compared to control, for this measure. Further comparison between the completers 
model and the imputed model suggested that the effects of the interventions were attenuated by the 
imputation. For instance, looking at the largest effect size for the intervention Arm 7 (CEM booklet and 
advanced CPD), the raw coefficient reduced from 0.78 in the completers model to 0.56 in the imputed 
model. For Measure 2, these results are presented in Table 6 and Figure 4b. 


Table 6: Secondary outcomes analysis, Measures 1 and 2—effect size versus control 
(Dunnett’s multi-level confidence intervals; numbers of teachers)'® 


IEE materials IEE evidence fair How2s website Laken] 0) ole) a -Lale| CEM booklet 
(Arm 1) (Arm 2) access (Arm 3) website (Arm 4) (Arm 5) 
n ES n ES n ES n ES n ES 
Outcome (missing) (95% Cl) | (missing) (95% Cl) (missing) (95% Cl) (missing) (95% Cl) (missing) (95% Cl) 

0.19 0.1 0.19 0.12 -0.24 
Measure1 | 22(9) | (-0.31, | 20(6) | (-0.42, | 23 (15) (- 16 (14) (-0.46, 14 (15) (-0.85, 
0.69) 0.62) 0.3,0.68) 0.69) 0.37) 
0.38 0.38 -0.06 0.18 -0.17 
Measure 2 22 (9) (-0.17, 20 (6) (0.18, | 23 (15) (-0.59, 16 (14) (-0 AA 0.8) 14 (15) (-0.83, 
0.92) 0.95) 0.47) es 0.48) 


ResearchEd 
CEM booklet and (oN efoto) «(=i ar-Lale| ResearchEd conference and 
light CPD EVe ME-Laler-\e I @7 od B) conference online community Control 
(UG) (Arm 7) (Arm 8) (Arm 9) (Arm 10) 


n =ts) n =ts) n ES n is) n 
Outcome ' (missing) | (95% Cl) (missing) (95% Cl) (missing) (95% Cl) (missing) (95% Cl) (missing) 


0.11 0.16 -0.03 0.21 
Measure1  21(7) | (-0.4,  20(9) | (-0.36, | 14(4) | (-0.64, | 13(10) | (-0.43, 172 (133) 
0.62) 0.68) 0.58) 0.84) 
0.39 0.54 -0.06 0.53 
Measure2  21(7) | (-0.16, | 20(9) | (-0.03, | 14(4) | (-0.72, | 13(10) | (-0.15, 172 (133) 
0.94) 1.11) 0.59) 1.21) 


Source: NFER Literacy Octopus Evidence-based Literacy Support Trial, 2016. 


Figures 4a and 4b show the effect size between the control arm and intervention arms for Measures 1 
and 2 (respectively). As discussed above, it can be seen from the charts that the confidence intervals 
straddle zero and therefore that there is no difference in the effect size of the control arm and any of 
the intervention arms for either of the measures. While there appear to be some variations in effect 
size across the intervention arms, it was beyond the scope of the current analysis to investigate this 
more thoroughly. 


16 The n for each arm is those with both baseline and follow-up data (across all arms, n = 335) and number of 
missing is those who were only found to be at end-point (across all arms 557-335 = 222). 
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Figure 4a: Secondary outcomes analysis, Measure 1—effect size versus control (Dunnett’s 
multi-level confidence intervals) 


Effect size between control group 
and trial arm for positive disposition 
to academic research in informing 
teaching practice (measure 1) 


IEE materials —_-+—__) 
IEE evidence fair ———$—=$= $<) 
How2s website access (——.+—_—_} 


How2s support and website ——_ +) 
CEM booklet. (--———e——_} 
CEM booklet and light CPD —-—_—_} 
CEM booklet and advanced CPD +; 
ResearchEd conference (——————} 


ResearchEd conference and online community ———— — 


-0.90 -0.45 0.00 045 0.90 


Effect size 


Figure 4b: Secondary outcomes analysis, Measure 2—effect size versus control (Dunnett’s 
multi-level confidence intervals) 


Effect size between control group 
and trial arm for uses 
of academic research to inform selection 
of teaching approaches (measure 2) 


IEE materials +—-—_) 

IEE evidence fair qe) 
How2s website access (—+—_ 

How2s support and website (—+—_) 

CEM booklet (e+ } 

CEM booklet and light CPD t++—-—_} 

CEM booklet and advanced CPD (——+—_; 
ResearchEd conference (—_-|__} 


ResearchEd conference and online community —— 
-1.30 0.00 1.30 
Effect size 
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Measure 1 was scored on a scale of 6—29 with a higher score reflecting a more positive disposition to 
academic research. Table 7 below displays the raw mean scores for Measure 1 and shows that 
across all arms and the control, teachers score highly on this Measure, indicating their positive 
disposition to research. 


Measure 2 was scored on a scale of 2-9 with a higher score reflecting greater use of academic 
research to inform teaching. Table 7 below displays the raw mean scores for Measure 2 and shows 
that across all arms and the control, teachers score slightly below the middle point of this measure. 
This indicates that, while teachers may have a highly positive disposition to research, it is actually 
used to inform the selection of teaching approaches to a lesser extent. 


Table 7: Secondary outcomes analysis, Measures 1 and 2—raw means (unadjusted confidence 
intervals; numbers of teachers at follow-up) 


How2s support 


IEE materials IEE evidence fair How2s website and website CEM booklet 
(Arm 1) (CNG) access (Arm 3) (Arm 4) (CNG) 
Mean Mean Mean Mean Mean 
Outcome (95% Cl) (95% Cl) (95% Cl) (95% Cl) (95% Cl) 
21.6 21.04 21.21 21.8 21.24 
Measure 1 31 (20.44, 26 19.69, 38 (19.94, 30 (20.65, 29 (19.82, 
22.75) 22.39) 22.48) 22.95) 22.66) 
4.77 4.58 4.47 51 4.41 
Measure 2 31 (4.13, 26 (4.03, 38 (4.05, 30 (4.48, 29 (3.93, 
5.42) 5.13) 4.90) 5.72) 4.9) 
ResearchEd 
(e431 efoto) d{=t ar-lale| CEM booklet and ResearchEd conference and 
light CPD EVeMVE-later=\e I @] oJ B) conference online community Control 
(Arm 6) (Arm 7) (Arm 8) Ue) (Arm 10) 
Mea Mean Mean Mean Mean 
Outcome (95% Cl) (95% Cl) (95% Cl) n (95% Cl) (95% Cl) 
21.61 21.72 21 21.85 21.66 
Measure 1 28 (20.52, 28 (20.4, 23.05) 18 (19.12, 23 (20.32, 305 (21.31, 
22.7) eee 22.88) 23.37) 22.01) 
5 5.03 4.56 4.83 4.6 
Measure 2 28 (4.43, 29 (4.39, 18 (3.81, 23 (4.06, 305 (4.45, 
5.57) 5.68) 5.30) 5.59) 4.74) 


Source: NFER Literacy Octopus Evidence-based Literacy Support Trial, 2016. 


The remaining outcome measures (3-6) from the Research Use Outcomes Survey were also 
included in the analysis. These were analysed at the school-level using ANOVA using the endpoint 
survey responses. This enabled us to explore any differences between the Literacy Octopus 
intervention arms and the control arm. 


The results of the ANOVA for Measure 3 (‘perception that academic research is not useful in 
learning’) revealed that the differences between any of the intervention arms compared to the control 
arm were probably due to chance (F = 0.544, df = 9, p = 0.842). Measure 3 was scored on a scale of 
2-10; the scale was negatively worded so scores at the lower end of the scale indicate a more 
positive disposition. Table 8 below displays the raw means for Measure 3 and shows that across all 
groups of respondents, teachers tend to be reasonably positive about the usefulness of academic 
research in learning. 


The results of the ANOVA for Measure 4 (‘perception that own school does not encourage use of 
academic research’) indicate that the differences between any of the intervention arms compared to 
the control arm were probably due to chance (F = 0.458, df = 9, p = 0.902). Measure 4 was scored on 
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a scale of 2-10, again, negatively worded: scores at the higher end of the scale indicate a more 
negative disposition. The mean scores for this measure are presented in Table 8 and show that 
across all groups of respondents, teachers are slightly positive about their schools’ encouragement 
for the use of academic research. 


The results of the ANOVA for Measure 5 (‘active engagement with online evidence platforms’) 
indicate that the differences in the mean scores for the randomisation arms were probably by chance 
(F = 0.892, df = 9, p = 0.532). Measure 5 was scored on a scale of 2—7; a higher score indicates more 
engagement with online evidence platforms. The mean scores for this measure are presented in 
Table 8 and show that across all groups of respondents, teachers tend to score slightly above the 
middle of the scale, indicating that they engage with online evidence platforms to a moderate extent. 


The results of the ANOVA for Measure 6 (‘your knowledge about research’) suggest that there is a 
difference between the mean scores between the randomisation arms (F = 1.996, df = 9, p = 0.039). 
Since this difference was found to be statistically significant, we ran multiple comparisons using the 
two-sided Dunnett’s Test. This post-hoc test compared each intervention arm with the control arm as 
a reference category. P-values for each comparison were found to be greater than 0.05 suggesting 
that there is no evidence that the mean score for each intervention arm is different from that of the 
control arm. Measure 6 was scored on a scale of 0-11; this measure is scored on the number of 
correct answers given to factual questions about research. The mean scores for this measure are 
presented in Table 8 and show that all groups of respondents tended to score fairly low on the scale, 
indicating a moderate level of knowledge about research. 


Table 8: Secondary outcomes analysis, Measures 3-6—school means 


How2s website How2s support and 
IEE materials IEE evidence fair access website CEM booklet 


(Arm 1) NW) NG es)) (Arm 4) (Arm 5) 


Mean Mean Mean Mean Mean 
Outcome n (95% Cl) (95% Cl) (95% Cl) (95% Cl) (95% Cl) 


4.46 


(3.85- 
5.07) 


5.31 


(4.54— 
6.09) 


CEM booklet CEM booklet ResearchEd eonicrencannd 
and light CPD and advanced CPD conference online community Control 
(Arm 6) (Arm 7) (Arm 8) (NG) (Arm 10) 


ResearchEd 


Mean (95% Mean Mean (95% Mean (95% Mean 
Outcome n (7) n (95% Cl) n ie) n (e7)) n (95% Cl) 
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Measure 4 


Measure 6 


Source: NFER ANOVA—Research Use Outcomes Survey analysis (Spring/Summer 2016 survey). 


Costs 


Table 9 presents details of how much it would cost a school to participate in each of the Literacy 
Octopus intervention arms. The table presents the cost per pupil per year over three years, as 
described in the EEF cost-evaluation guidance (and is accompanied by notes explaining the 
assumptions underpinning these calculations). The table also summarises any prerequisite resources 
and requisite staff time associated with participating in the interventions in 2015, and in any ongoing 
engagement in 2016 and 2017. As the information on the costs to schools of taking part in the 
interventions is based on a small number of cases, it provides indicative information only. 


Following Table 9, we briefly describe how much the Literacy Octopus interventions actually cost 
schools which took part in the evaluation, and explore case-study schools’ reports of their financial 
and time investment in these interventions. We have also included some description from case-study 
schools of the time they invested in resultant implementation activity. 
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Table 9: Summary of costs to schools 


1. IEE: 
materials 


2. IEE: 
evidence fairi 


Year 1 
(2015) 


Year 2 
(2016) 
Year 3 
(2017) 


Year 1 


Year 2 
Year 3 


Subscription fees: £19.50 per person per 
year or £50 per institution per year.? 


As previous year. 
As previous year. 


Subscription fees: £19.50 per person per 
year or £50 per institution per year; 
event fees: no charge; 
travel to event, two members of staff to 
attend: £200.°¢ 


Subscription fees: £19.50 per person per 
year or £50 per institution per year 
As Year 2. 
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Computer with 
internet. 


Computer with 
internet. 


Intervention engagement: small amount of time to read 
materials, e.g. 1 hour per month per person; 
staff preparation time (reviewing and planning approaches): 
e.g. 2 hrs. 
Ongoing engagement: small amount of time to read 
materials, e.g. 1 hour per month per person. 


As Year 2. 


Intervention engagement: small amount of time to read 
materials, e.g. 1 hour per month per person; 
attending event: 1 day per person (2 days total); 
supply cover time/costs: as applicable; 
staff preparation time (reviewing and planning approaches): 
e.g. 2 hrs. 


As Year 2, Arm 2. 


As previous year. 


£0.31 


£0.734 


3. Teaching 
How2s: 
website 
access 


4. Teaching 
How2s: 
support and 
website 
access 


Year 1 


Year 2 
Year 3 


Year 1 


Subscription fees: £624 (inc. VAT) per 
school per year.® 


As previous year. 


As previous year. 


Subscription fees: £624 (inc. VAT) per 
school per year; 
travel to introductory event, one member 
of staff to attend: £100. 
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Computer with 
internet. 


Computer with 
internet. 


Intervention engagement: small amount of time to initially 
acccess and use website, e.g. 2 hours per person; 
build into existing CPD: e.g. 0.5 day per school. 
Ongoing engagement: small amount of time to continue to 
build into existing CPD: e.g. 0.5 day per school. 

As previous year. 

Intervention engagement, attend introductory event: 1 day 
per person (1 day total); 
supply cover time/costs: as applicable; 
small amount of time to initially access and use website: e.g. 
2 hours per person; 
in-house cascaded CPD: e.g. 1 hour; 
staff preparation time (reviewing and planning approaches): 
e.g. 2 hours; 
coaching observations: e.g. (3 x 15 mins to 1 hr 


£3.90 


£4.11 
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5. CEM: 
booklet! 


6. CEM: 
booklet and 
light CPD 


Year 2 


Year 1 


Year 2 
Year 3 


Year 1 


Year 2 
Year 3 


Subscription fees: £624 (inc. VAT) per 
school per year. 
As year 2. 


Printing: booklet, £2 per booklet; 
posters, £1 per poster x 12 per year. 


As previous year. 


As previous year. 


Printing costs (as Arm 5) plus 
training fees (one event): £54 for 3 
teachers to attend;' 
travel to training (one event): £90 for 3 
teachers to attend.9 


Printing costs only (as above). 


Printing costs only (as above). 


Printing costs (as above) plus 
training fees (two events): £108 for 3 
teachers to attend; 
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Computer with 
internet; 
printer. 


Computer with 
internet; 
printer; 
resources (e.g. 
workbooks). 


observations); 

build into existing CPD: e.g. 0.5 day per school. 

Ongoing engagement: small amount of time to continue to 
build into existing CPD: e.g. 0.5 day per school. 

As previous year. 
Intervention engagement, staff preparation time (reading 
booklet): e.g. 1.5 hrs per person; 
staff preparation time (reviewing and planning approaches, 
using posters): e.g. 2 hrs. 

Ongoing engagement: small amount of time to continue to 
read/use materials: e.g. 0.5 hour per month per person. 
As previous year. 

Intervention engagement, attend training: 2 hrs per person (6 
hrs total); 
staff preparation time (reading booklet): e.g. 1.5 hr per 
teacher; 
staff preparation time (reviewing and planning approaches, 
using posters): e.g. 2 hrs per person; 

in-house cascaded CPD: e.g. 2 hrs per school. 


Ongoing engagement: staff preparation time to read/use 
materials: e.g. 0.5 hour per month. 
As previous year. 
Intervention engagement, attend training: 2 x 2 hr sessions 
per person (12 hrs total); 
INCAS peer observations: e.g. 2 hrs per school; 
staff preparation time (reading booklet): e.g. 1.5 hr per 


£0.09 


£0.39 


7. CEM: Year 1 | travel to training (two events): £180 for 3 | Computer with teacher: 
striae teacie ® 10 allenic internet; staff preparation time (using posters reviewing and planning 
advanced INCAS subscription: £10.08 (inc. VAT) printer; ties 
CPD arpupill tt strategies): e.g. 2 hrs per person; 
perpePe anener (e.g. in-house cascaded CPD: e.g. 2 hrs per school. 
—_ PowerPoint). ; paper £10.77 
Printing costs (as above) plus INCAS Ongoing engagemert: staff preparation time (to read/use 
Year 2 cae wow 
subscription (as above). materials: e.g. 0.5 hour per month per person. 
Year 3 As Year 2. As previous year. 
r Intervention engagement, attend event: 1 day per person; 
8. : i 
ResearchEd: Ey aH Ce> or teu naven tan ae) Resoures (e.g. in-house cascaded CPD: e.g. 1 hr; 
Year 1 travel to event, one member of staff wo gs £0.26 
conference ais books). staff preparation time (planning and implementing approach, 
attending: £100. ; : 
coaching colleagues): e.g. 1 day. 
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conference 
and online 
community 
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Year 2 None. 
Year 3 None. 
Year 1 As Year 1, Arm 8. 
Year 2 None. 
Year 3 None. 
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Computer with 
internet; 
resources (e.g. 
books). 


Ongoing engagment: none/minimal. 


As previous year. 
Intervention engagement, attend event: 1 day per person; 
attend webinars: 2 x 1 hr sessions per person; 
in-house cascaded CPD: e.g. 1 hr; 
Staff preparation time (implementing approach, coaching 
colleagues): e.g. 1 day. 


Ongoing engagement: none/minimal. 


As previous year. 


£0.26 
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Notes on Table 9 


4 For the purposes of the per-pupil calculation, we have assumed the subscription cost would be £50 
for the whole institution. As the trial was focused on literacy across KS2, we have assumed that a 
typical primary school in the trial sample will have more than one teacher of this Key Stage. 


> The per-pupil calculations are based on the marginal costs detailed in the table. The calculation is 
based on: total cost of the intervention over three years; divided by three; divided by 160 pupils in 
KS2. For the purposes of the calculation, we calculated an average number of pupils in KS2 primary 
schools to be 160 based on the sample of primary schools participating in the trial (four year groups, 
with an average 40 pupils per year group). 


© Travel costs varied depending on the mode of travel and proximity to venues. In some cases, other 
costs relating to travel may be incurred, such as an overnight stay to enable timely arrival at the 
venue and subsistence costs during travel. To take account of these costs we have assumed a typical 
cost of £100 per person per return journey to events, unless otherwise stated. 


4 The per-pupil calculation includes travel costs for two members of staff to attend this event as IEE 
encouraged a senior leader and class teacher to attend from each school. In all other cases, unless 
stated otherwise, per-pupil calculation costs are based on one member of staff attending 
events/training (who may subsequently cascade the information to their colleagues within school as 
part of general CPD activity). 


© The subscription fee was waived for Literacy Octopus trial schools in the first and second years of 
the trial (2014/2015 and 2015/2016). Trial schools were subsequently offered the annual subscription 
at £520 + VAT per year, or a discounted rate if they subscribed for two years to get a free third year, 
totalling £1,040 + VAT for three years subscription. The subscription structure varies with the size of 
the school. For the purposes of the per pupil calculation, we have assumed an annual subscription 
charge of £520 + VAT (£624). 


‘ We were informed by the provider that the costs of the training varied according to the event location 
and number of teachers attending. Venue costs were £130 outside of London, £500 within London, 
and £450 to buy in training delivered at a host school. This pricing structure is based on sharing these 
costs among an assumed 20 teachers attending. This gives an average event cost of £360 and an 
average cost per teacher (assuming 20 attend) of £18. We were informed by the provider that, on 
average, two to four teachers from a school attended training and, hence, for the calculation we have 
assumed that an average of three teachers attended each training event. 


3 Events run by CEM tended to be small, localised events (often hosted in participating schools) and 
so travel costs to the training tended to be low and were estimated by CEM to average £30 per 
teacher. Arm 6 involved one twilight training event, while Arm 7 involved two twilight training events 
and hence the average cost of travel per teacher has been doubled in the per-pupil cost for Arm 7. 
We have estimated the per-pupil cost based on three teachers from a school attending the training 
event/s. 


h We were informed by the provider that the cost of INCAS was £8.40 per pupil plus VAT. These 
charges were waived for trial schools during the first year of the trial (2015). 


i The cost of attending ResearchEd conferences varies depending on the location, numbers attending, 
and sponsorship funding for each event. We were informed by the provider that the average event 
charge is £25. This charge was reduced to £15 for teachers attending the events as part of the 
Literacy Octopus trial (2015) and was reimbursed upon attendance. 


i We were not able to collect cost information directly for Arms 2 and 5 as we were unable to recruit 
case-study participants for these arms. In these cases, we have provided a cost estimate based on 
supplementary information from schools regarding other arms and similar activities, as well as from 
the providers. 
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The following paragraphs briefly describe in turn how much each Literacy Octopus intervention cost 
for the schools that took part in the evaluation. It also explains the views of case-study schools on 
whether any additional costs were incurred or additional time required to participate in the 
interventions and any resultant implementation activity. 


Costs of participating: IEE materials and evidence fair 


Schools participating in these interventions were not charged any subscription fees for paid-for 
materials (some of the IEE materials are freely available to schools anyway). The evidence fair was 
free to participating schools. A case-study school reported that there were no additional costs, 
resources, or time-demands associated with engaging with the materials, as reading and discussing 
them was considered part of normal CPD activity (such as discussing ideas at staff meetings) and any 
resulting modifications to practice used existing resources. Supply cover costs could be incurred by 
schools to attend events, and were seen as additional or not, as applicable to each school’s supply 
cover practices. 


Costs of participating: Teaching How2s teaching and learning website and introductory event 


Schools participating in these interventions had free access to the Teaching How2s website during 
the first and second years of the evaluation (spring and summer terms of 2015, and academic year 
2015/2016). The introductory event was also free to schools. Subsequent to this evaluation, schools 
faced the normal subscription charges to continue to use the website if they wished, but with a 
discounted rate of ‘buy two years, get three’ as described in the notes above. Case-study schools 
expressed concern about meeting this cost and suggested that it may prohibit their longer-term use of 
the resource. Both case-study schools were exploring the possibility of offsetting the cost of 
subscription by becoming How2s ‘hub’ schools and supporting neighbouring schools to use the 
resource. Participating schools identified the cost of travel to the introductory event as additional to 
normal expenditure. The schools did not have to purchase any additional resources to use Howes. 
Engaging with this intervention required teachers to spend time attending the induction, cascading 
information to colleagues, planning how to implement ideas, and conducting peer-observations. 
However, this was generally regarded as being commensurate with normal teaching and teacher- 
development activities, and though it may have displaced time spent on other development activities 
with a different focus, overall, it required little additional time investment. Using the How2s website as 
part of the trial was considered by case-study interviewees as a relatively low-cost intervention, 
particularly given it is school-based (but note, the subscription fee was waived for the trial). 


Costs of participating: CEM materials, training and INCAS diagnostic tool 


Schools participating in these interventions received the printed booklet and posters free in the post 
(so did not incur printing costs) during the first year of the evaluation (spring and summer terms 
2015). The training events and access to INCAS were also free to schools during the first year of the 
evaluation (2015/2016). The case-study schools reported either no, or very minimal, travel costs to 
attend the training events as these were either hosted by the school or were very local. Schools did 
not have to purchase any additional resources as part of the intervention as existing resources could 
be used (such as a computer with internet access, software such as PowerPoint, printing and 
laminating, and workbooks). Engaging with this intervention required teachers to spend time attending 
training, cascading information to colleagues, discussing and reviewing approaches at staff meetings, 
planning how to implement ideas, and creating and collating teaching and learning resources. 
Interviewees views varied as to whether this time was considered as additional—the twilight CPD 
sessions were considered as additional time by one teacher—though generally these activities were 
considered to be a reasonable investment of time and part of normal practice for teachers. 
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Costs of participating: ResearchEd and NatCen conference and webinars 


Schools participating in these interventions paid a nominal charge of £15 to attend a conference 
during the first year of the evaluation (2015/2016) which was reimbursed upon attendance. The pre- 
and post-conference online webinars were free for participating schools to access. The case-study 
schools incurred small additional costs for travel to attend a conference. Schools did not have to 
purchase any additional resources to engage with the intervention itself as these already existed in 
the school (for example, a computer with internet access). However, case-study schools did incur 
additional costs associated with implementing some of the practices introduced at the conference 
(such as purchasing books suitable for reciprocal reading, buying an additional external training day 
on Fischer Family Trust reciprocal reading, or buying a one-to-one phonics tutoring kit, workbooks, or 
phonics-based games). Engaging with the Literacy Octopus intervention itself required a day of 
additional time to attend the conference (this was on a Saturday so was outside of normal working 
hours, though some schools gave teachers a working-day off in lieu) and online webinars. Teachers 
also reported that implementing new teaching strategies identified at the conference also involved 
their time to: cascade information to colleagues; attend additional training on the approach; plan how 
to implement the approach and trial in class; create and collate resources; and coach colleagues on 
using the new approach. Most of these tasks were considered to be part of normal teaching activity, 
and although this may have displaced a focus on a different approach, this did not represent 
substantial additional time burden. 


Education Endowment Foundation 47 


Evidence-based Literacy Support — the ‘Literacy Octopus’ Trial 


Process evaluation 


Summary of Process Evaluation findings 


Engagement: schools’ engagement with the Literacy Octopus interventions varied greatly: 42% of 
the 540 intervention schools did not engage with the Literacy Octopus materials according to 
monitoring data; 20% engaged a little; 22% engaged to the level planned for their arm; and 16% 
engaged to a greater extent than expected (for example, they hosted CPD sessions, or they 
requested further materials). Engagement also varied across the arms, with greatest engagement in 
Arms 1, 4, 5 and 6 (IEE passive, How2s active, CEM passive, and CEM active light). Least 
engagement occurred in Arms 8 and 9 (ResearchEd passive and active). 


In terms of IPE outcome survey respondents, around a half of intervention respondents reported 
that they had been directly or indirectly (for example, through information received from colleagues) 
involved with the Literacy Octopus (the view of 52% (120) of 234 respondents who were shown this 
question). These respondents were routed on to answer more detailed questions about 
implementation. The remaining respondents either had not been involved or did not know. 


Communication approaches: teachers were generally positive about the modes of communication 
used by the Literacy Octopus interventions and felt the amount of information provided was about 
right (70% of 120 survey respondents). Teachers showed no clear preference for a particular method 
—they engaged with both passive and active dissemination approaches—but wanted communication 
to be accessible and timely. 


Research evidence nature: case study and survey responses indicated that research evidence is 
most effectively communicated when it is interactive, accessible, relevant to enhancing and 
developing practice, supports the implementation process, includes a balanced and credible 
discussion of evidence, and focuses on how to apply the evidence to practice. 


Perceived outcomes and impacts: the majority of the 120 teachers responding to survey questions 
about perceived outcomes thought that participating in the Literacy Octopus had led to positive, 
practical impacts—such as enabling them to reflect on classroom practice (36% ‘a lot’, 56% ‘a little’), 
reinforce it (82% ‘a lot’, 57% ‘a little’), or change it (24% ‘a lot’, 56% ‘a little’), or to discuss best 
practice with colleagues in their school (32% ‘a lot’, 53% ‘a little’). 


Mechanisms for further implementation: where schools had gone on to implement changes to KS2 
literacy or research use, such changes were facilitated by activities such as: in-school collaboration; 
undertaking further enquiry and follow-up activities with providers; trying out, reviewing, adapting and 
embedding approaches; consulting research champions; and having a planned approach to 
implementation. Appendix | presents examples of impact trails in four schools. 


Fidelity: while the core communication mode and content in each intervention arm was delivered as 
planned, providers adapted their implementation in two main ways: (1) all providers found the initial 
response to advertised events lower than expected and so put in place additional measures to 
encourage attendance, and (2) a number of providers delivered alternative ‘active’ sessions, for 
example webinars, where the planned in-person dates and venues were not convenient to teachers. 


Formative findings: for future development, organisations involved in supporting teachers’ evidence- 
use need to consider whether to put efforts into increasing initial engagement (for example, through 
greater lead-in time, local rather than regional events, or hub-led events), and/or into transformation 
and implementation (through collaborative, planned, and championed approaches). It was found that 
one-third of survey respondents would not recommend the Literacy Octopus intervention they were 
assigned to, mainly due to a perceived lack of fit with their school’s current priorities, hence initial 
engagement and matching of evidence foci to schools’ needs (rather than the random allocation 
tested in this trial) may be an important consideration. 


Conirol group activity: control group schools were positive about the support they had received for 
KS2 literacy from external providers (not involved in the Literacy Octopus) during the trial period and 
indicated varying levels of research engagement. We found no evidence of resentful demoralisation 
or compensation rivalry among control schools and they appeared to continue with ‘business as 
usual’ in terms of their literacy practices and research engagement during the trial. 
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This section draws on data from the process evaluation, including: 


e the teacher outcomes surveys for the intervention and control groups (N = 252 and 305 
respectively); 


e early telephone interviews with 29 teachers; 

e case study interviews with 21 teachers in ten schools;'” 

e six interviews with control group teachers; 

e interviews with providers in 2014, 2015, and 2016 and their ToCs; 
e observations of events and review of materials; and 


e engagement data provided by Literacy Octopus providers. 
Levels of engagement with the interventions 


Literacy Octopus providers supplied information to NFER on schools’ engagement with the resources 
and activities in each arm using a monitoring specification developed by NFER (see Methodology for 
further details). NFER then applied a coding scheme to categorise schools’ engagement levels as 
‘high’, ‘mid’, ‘low’, or ‘non’-engaged. These schemes were based broadly on levels of engagement 
being ‘none at all’, ‘less than expected’, ‘as expected’, or ‘more than expected’ for each arm. For each 
arm, the expected level of engagement (for the activity that teachers were invited to engage in) was 
categorised as the ‘mid’ or ‘norm’ in order to provide some comparability across arms. Both teacher- 
and school-level monitoring were taken into account to categorise level of engagement per school 
(see Appendix F for details of the coding schemes). 


Fig. 5: Overall level of school engagement in Literacy Octopus as at July 2015 


As Figure 5 shows, schools’ overall 
levels of engagement varied greatly: 
42% of the 540 intervention schools 
did not engage with the Literacy 
Octopus materials, according to 
monitoring data; 20% engaged a 
little; 22% engaged to the level 
planned for their arm; and 16% 
engaged to a greater extent than 
expected (for example, they hosted 
Low CPD sessions, or they requested 
further materials).18 
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Source: NFER Literacy Octopus monitoring 
data. 


As shown in Figure 6, level of engagement varied across the arms, with greatest engagement (the 
highest ‘mid’ and ‘high’ levels) in Arms 1, 4, 5 and 6 (How2s active, IEE passive, CEM passive and 
CEM active light) and least in Arms 8 and 9 (ResearchEd passive and active). Interestingly, it was not 
necessarily the case that the more active the arm, the higher the schools’ engagement. 


17 By ‘teachers’ we mean class teachers, school leaders, and literacy co-ordinators in primary schools. Where 
different respondents contributed differently, we identify their job role. 

18 For Arm 8, the criteria for categorising ‘none’ and ‘low’ engagement were the same (that is, if no teachers from 
the school attended the conference, for analysis purposes, these were categorised as ‘none’). 
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Providers supplied a further round of monitoring data in April 2016 to help measure longer-term 
activity and engagement. Most schools remained at the same level of engagement as in the previous 
year, although some levels changed in either direction reflecting schools’ changeable circumstances 
and priorities. 


Senior leaders’ and teachers’ reasons for taking part in the Literacy Octopus and engaging with their 
arm focused mainly on wanting to improve KS2 literacy—to benefit from what they felt would be the 
latest expertise in literacy support without themselves having to spend time finding those resources or 
assessing the relevance and accessibility of the materials or activities. Reasons for not engaging 
tended to relate to a lack of time, the timing and location of events, a perceived lack of relevance of 
the content of the materials or activities, and a preference for an alternative mode of communication 
to that offered. 


Fig. 6: Level of engagement as at July 2015 
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Source: NFER Literacy Octopus monitoring data. 
Numbers per arm may not sum to 60 as in some arms schools have withdrawn from the trial and their data being used. 


In addition to the MI data, we also asked survey respondents about the extent to which they had 
taken part in the Literacy Octopus arm they were allocated to. Table 10 shows these responses and 
that around a half of intervention respondents reported that they had been directly or indirectly 
(through colleagues) involved—52% of 234 respondents who were shown this question. These 120 
respondents were routed on to answer more detailed questions about implementation. The remaining 
respondents either had not been involved or did not know. 
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Table 10: Extent of school involvement in Literacy Octopus according to survey respondents 


Did your school take part in the Literacy Octopus arm you were % (N) (N = 234) 
F1| fofer 1 (10 cog 

Yes, and | had first hand involvement. 34% (79) 
Yes, my colleague/s were involved, and they shared the learning with me. 18% (41) 
Yes, my colleague/s were involved, but | don’t know any more about it. 11% (26) 
No, my school did not take part. 18% (41) 
I’m not sure. 20% (46) 
Missing. 0% (1) 


A series of single response items. 
A total of 252 teachers completed the intervention survey. A total of 234 teachers saw this question; 233 gave a response. 
Source: NFER Literacy Octopus teacher outcomes survey, intervention arms, 2016. 


Implementation 


Communication approaches 


Teachers were generally positive about the modes of communication used by the Literacy Octopus 
interventions and felt the amount of information provided was ‘about right’ (70% of 120 survey 
respondents). Teachers showed no clear preference for a particular method (they engaged with both 
passive and active dissemination approaches) indicating that a multitude of methods is required to 
allow teachers to engage in a way that best suits their needs. Table 11 shows that teachers were 
generally positive about all forms of communication of the materials—noticeably more teachers 
indicated that the mode of communication was either ‘very good’ or ‘quite good’, than those rating 
‘quite poor or ‘very poor’. 


Table 11: Teachers’ views on the modes of communication used Literacy Octopus providers 


How would yourate the Very Quite | Average Quite Very Did not Total 
following elements of fe Lozeye| fe foreye| Xero) g jeXere) g use Lc=X-) oe ate llale] 
the LO materials and % % % % % % to item 
JU] o) oe) a wan (Ch k:)) NY 
Posted materials 24 33 N=116 
Electronic materials 23 27 15 3 2 30 N=115 
Websites 18 22 18 1 1 40 N= 114 
SO ee 7 9 8 2 2 72 N=112 
fairs/conferences 

CPD/training sessions 17 11 10 3 1 59 N=113 
Online events e.g. 5 9 5 D 3 76 N=114 
webinars 


Teacher 


observation/teacher peer 
learning 


Pupil assessment tools 6 13 7 4 1 68 N=113 


A series of single response items. 


A total of 252 teachers completed the intervention survey. A total of 120 teachers saw this question; 117 gave at least one 
response and 3 did not respond. A filter was applied to this question to only analyse responses from teachers who indicated 
that they had direct involvement with the Literacy Octopus. 


Source: NFER Literacy Octopus teacher outcomes survey, intervention arms, 2016 
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Teachers reported that ‘effective’ and ‘very effective’ modes of communication related to opportunities 
for face-to-face contact and direct communication and support from the providers and other teachers. 
Some form of ongoing communication (as opposed to one-off) was also reported by teachers and 
providers to be effective, for instance a series of activities with preparation and follow-up or regularly 
communicated materials. Location and timing of activities were also important factors for teachers; 
events needed to be proximate and at accessible times. Access to hard copies of materials was also 
valued by teachers. 


The nature of the research evidence 


Interviewed teachers were divided in their views on the usefulness of the content of the materials and 
support they had received; a similar proportion found them useful as those who found them not 
useful. Teachers found the materials and activities useful when relevant to their practice (for example, 
a breadth and variety of content, innovative ideas and strategies, and details of how strategies could 
be applied and adapted in practice). Teachers also said it was important that content was presented 
in an accessible, engaging, and easy-to-follow style. 


Teachers and providers suggested that the materials and activities needed a balanced focus on 
evidence, with explicit reference to evidence on the impact of the strategy and how to effectively 
implement it, yet without overloading the user by providing too much detail about individual studies. 
This headteacher explained: 


It gives you evidence, and you can’t dispute evidence; it gives you some onus and impetus 
that it’s not just a whim. The fact that there is evidence behind this, there is a reason, so staff 
are more like ‘oh right, we'll give that a go’. 


As well as empirical research, teachers also clearly valued teacher endorsement and practical 
examples of how teaching strategies had been effectively used. 


Mechanisms for successful implementation of research-evidence in schools 


The process evaluation also explored how teachers and school leaders implemented the strategies 
presented within the Literacy Octopus materials and support in their schools. Although overall we 
found no evidence of the impact of the Literacy Octopus arms on schools, we did find a small number 
of schools that had successfully implemented the evidence-based strategies. Analysis suggested that 
in these cases there were a range of facilitating mechanisms (and conversely, the absence of such 
mechanisms inhibited the impact). These mechanisms pertain to both the provider (such as how the 
information and support is provided; the content of the materials and support; and the ‘transformation’ 
of evidence for practice) and the school (how they engage with the evidence, the implementation 
processes, and pre-existing contextual and cultural factors). The mechanisms are summarised in 
Table 12 below in the order of frequency with which they were identified in the analysis. 
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Table 12: Mechanisms used in the provision of Literacy Octopus interventions 


Mechanism 


wd enva(e (=) moze) aly (e(sVe-lilelalsy 


Cde-Talyce)guat-1i(o)am-| e)e)ger-\elal=y-)) 


sXe) YoYo) Moxey at-y(o(=Ve-1i(o) aly 
(implementation approaches) 


Relevance 


Provides relevant research evidence 
on topics of concern to schools and 
‘whether’ and ‘how’ a particular 
strategy works. Provides guidance 
and materials to support 
implementation in practice. 


Identifies areas for development and 
uses research to explore solutions. The 
evidence-based practice/approach fits 
with school priorities/values about 
teaching and learning. Culture of 
engagement with and valuing research 
evidence. Senior leadership endorses a 
focus on the strategy (as part of initial 
decision-making) and makes time and 
resources available to support 
implementation 


Collaborative 


Facilitates and promotes 
professional discussion and 
interaction, ideally face-to-face. 


Engages in professional discussion, 
collaborative decision-making, and 
sharing with colleagues through 
meetings, training, and other activities to 
disseminate research-informed 
practices. 


Status of Provides synthesised but clear and Considers and uses evidence to inform, 

evidence explicit research evidence and justify, or challenge the implementation 
discussion of theories underpinning of a strategy. 
the practices considered by credible 
experts, as well as examples from 
practice/practitioners. 

Further Provides ongoing (or initial and Undertakes further enquiry, information- 

enquiry and follow-up) input and contact, with gathering, and training (to extend 

follow-up further information, support and knowledge and deepen understanding) 
guidance. of the strategy. 

Embedded Encourages schools to consider how | Embeds evidence-based strategies 
strategies will be embedded long- within ongoing practices (e.g. within 
term. policy). 

Try-out and Encourages schools to try out and Tries out a new evidence-based 

review review strategies in practice and strategy in practice and reviews its 
shares insights from experiences of effectiveness to inform subsequent 
implementation across schools. action. 

Champion Encourages schools to identify Identifies an appropriate member of 
appropriate staff well-placed to staff, with enthusiasm and relevant 
engage with, support, and implement | knowledge and skills, to lead and 
the evidence-based strategies support the implementation process and 
considered. activities, and gives them capacity and 

authority to implement strategies. 

Adaptation Provides information on a range of Adapts strategies to meet school, 


approaches or considers issues of 
adaptation to practice to enable 
knowledge-users to customise 
information to their own needs. 
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Accessibility Offers accessible information, A lead member of staff engages with a 
different modes of engagement (e.g. | breadth of information and filters for 
printed materials, face-to-face colleagues to provide accessible 
training, webinars) and flexibility (e.g. | information. 
timing and location). Minimal cost for schools to implement. 


Minimal cost for schools to access. 


Planned Encourages schools to plan the Creates a formalised action plan to map 
implementation process step-by-step | out the route to implementation and 
and consider the resources required | resourcing required. 

in implementation. 


Source: NFER analysis of how providers transformed and teachers implemented Literacy Octopus evidence (NFER Literacy 
Octopus evaluation 2016) 

This typology builds on previous literature exploring effective knowledge mobilisation. For instance, 
Becheikh et al. (2009) also identified the relevance of knowledge and practitioners’ organisational 
culture as being important factors of effective knowledge mobilisation. The ‘collaborative’ mechanism 
reflects the importance of social interaction identified by others in the field (Durbin and Nelson, 2014). 
The ‘relevance’ mechanism highlights the value of research evidence that has been ‘transformed’ into 
implementable strategies, also highlighted by Nelson and O’Beirne (2014). In addition, these more 
interactive implementation approaches reflect the existing evidence that more socially interactive and 
collaborative forms of research engagement are more effective (Becheikh ef al., 2009; Nelson and 
O’Beirne, 2014; Sharples, 2013). In addition, having a research champion to drive the change was 
key in case-study schools (reflecting other recent literature: Lord et al., 2017 and Griggs et al., 2016). 
Detailed examples of how these mechanisms have facilitated the impact of the Literacy Octopus 
materials and support in four case-study schools are provided in Appendix |. 


Challenges and barriers 


In many cases, teachers reported challenges in engaging with, and using, the Literacy Octopus 
materials and support where the effective features and facilitating mechanisms, discussed throughout 
the sub-sections above, were not in place. In summary, the challenges and barriers that teachers 
experienced predominantly related to: 


e contextual factors within the school (such as a lack of time and the pressure of other 
priorities); 


e the mode of communication (such as inaccessible events); 


e the content of the materials and support (for example, not being relevant for experienced 
teachers or particular pupil groups; 


e limited evidence of the impact of the strategy in practice—and insufficient information to 
enable implementation in practice; and 


e implementation factors in school (such as staff changes, lack of senior management support 
and resources to implement new strategies, and colleagues’ reluctance to engage with the 
evidence or the strategy). 


In addition, all of the providers identified challenges in initially engaging schools. Often the providers’ 
initial email approaches were screened out as junk email or failed to reach the most appropriate 
member/s of staff in schools. Some providers felt that the focus of the materials and activities on Key 
Stage 2 literacy did not have sufficiently wide appeal to schools. Providers also identified issues with 
their profile and credibility with the Literacy Octopus sample of schools: where the provider was not 
known to the schools, some providers felt that the school was less likely to participate in the 
intervention offered. Providers also found it challenging to engage schools where they perceived there 
was no, or only a low level of, previous engagement with research evidence. 
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Fidelity 


While the core communication mode and content involved in each Literacy Octopus arm was 
delivered as planned, providers’ adapted their implementation in two main ways: 


e all providers found the initial take up of event places lower than expected and so put in place 
additional rounds of initial contacting and invitation to engage schools in events; and 


e anumber of providers delivered alternative ‘active’ sessions, for example webinars, where the 
planned in-person dates and venues were not convenient to teachers. 


Furthermore, some school staff who attended the events seemed to be uncertain about, or unaware 
of, their participation in a research study. Where appropriate, therefore, the events had to incorporate 
an extra session explaining the research study to the teachers (including, in some instances, support 
from NFER researchers) and helping them to understand why they were at the event. Below are 
individual issues with fidelity reported by the providers. 


The Institute for Effective Education at York University 


Interest in the activities on offer to schools in the active arm was lower than expected. As a 
consequence, the initial conference day had lower numbers than anticipated, and it was decided to 
cancel the post-conference webinar due to the lack of interest. Instead, all schools in IEE’s active arm 
were invited to another conference run by IEE in the summer, opened up to teachers outside of the 
research project to make sure that suitable numbers attended for the event to be viable. This second 
event also focused more on theory and evidence rather than a marketing pitch for individual 
interventions, as, through a round of follow-up telephone calls with schools, the providers realised that 
this aspect had not been received well by all at the initial conference day. 


Teaching How2s 


As well as the common issues mentioned above regarding low engagement and the need to provide 
delegates with further information about the research project at the training day, some schools in the 
active arm for this provider took part in the providers own case studies (separate from the 
independent NFER evaluation). Schools selected as case studies were visited by members of the 
Teaching How2s team and spent the day discussing how the How2s materials were currently being 
used in the school. This applied to only six schools in the active arm—those most engaged with the 
materials, and indeed also categorised as highly engaged by NFER according to monitoring data. 


Centre for Evaluation and Monitoring at Durham University 


While the active arms of CEM’s intervention experienced the same problems with low engagement, 
CEM felt that this was actually a benefit to those in the active arms, as the smaller than anticipated 
CPD sessions allowed teachers more time to share ideas with each other and made the sessions 
more interactive. 


The schools in the active group that offered a second CPD session were given the opportunity to 
either attend the second session in person, or in a revised approach to the original plan, to attend via 
a webinar. This was due to the large distances some schools would have had to have travelled to 
attend the second CPD session. 


ResearchEd and NatCen 


The take up for the pre- and post-conference webinars was very low. While the original intention had 
been to have separate webinars for those attending the south (London) and north (proposed location 
of Leeds, actual location York) conferences, for the post-conference webinar, only one was run to try 
and increase numbers—as the content of the two conferences had been identical. However, uptake 
remained very low for these, meaning that there was ultimately very little difference between the 
active and passive arms in terms of actual activity. 
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The low engagement, and therefore low numbers that signed up for the conference day, meant that 
extra schools were invited from outside the research project to both the conference days and the 
webinars to make them viable events. 


Perceived outcomes and impacts 


Perceived impact on teachers 


Although we found no effect of the Literacy Octopus interventions in terms of the outcome measures 
used for this study, the majority of the engaged teachers who completed our survey (N = 120) 
perceived that participating in the Literacy Octopus had led to positive impacts on their practice. Table 
13 displays these responses (the four strongest impacts are highlighted). 


Table 13: Teachers’ views of the impact of Literacy Octopus on practice 


To what extent did the information from the : Total 
LO arm you were allocated to enable you to... A lot A little Notatall responding 
(ede) 7) 7) to item (N) 
... discuss best practice with colleagues in my 32 53 15 118 
school 

... Share the learning with people or organisations 

outside my school ° a oo ae 

... reflect on my own practice 36 56 8 118 

... change classroom practice (this could be 

starting, developing or discontinuing an approach) ae ae ae me 

... reinforce existing practices 32 57 11 118 

... conduct my own research or enquiry 7 26 67 117 

... influence colleagues in my school to change 

their classroom practice (this could be starting, 24 41 35 116 
developing or discontinuing an approach) 

... improve my knowledge of KS2 literacy 15 52 33 116 


A series of single response items. 


A total of 252 teachers completed the intervention survey. A total of 120 teachers saw this question; 118 gave at least one 
response and 2 did not respond. A filter was applied to this question to only analyse responses from teachers who indicated 
that they had direct involvement with the Literacy Octopus. 


Source: NFER Literacy Octopus teacher outcomes survey, intervention arms, 2016. 


Perceived impact on literacy teaching 


Interviewed teachers provided examples of where engaging with the interventions had increased their 
awareness and understanding of literacy strategies, approaches, and practices. For instance, the 
interventions provided awareness of an alternative approach and perspective, as one headteacher 
put it: ‘it’s broadened people’s exposure to new strategies’. Often the strategies were familiar and 
served to remind, refresh, and reinforce existing thinking and practice, as one assistant headteacher 
explained: 


‘The [CEM] posters reinforce what we’ve been doing, particularly metaphors and riddles. They 
also talk about things we sometimes have less time to focus on, like anagrams and they 
reinforce vocabulary.’ 


The interventions helped teachers to understand more about how to effectively implement and refine 
literacy approaches, as this deputy headteacher describes: 


‘By going to the [CEM] CPD events the staff have more knowledge and can enthuse you to 
think about practical ways that you could introduce it and how other schools have done it.’ 
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Where mechanisms to support research engagement and implementation were in place (for example, 
there was relevance to an identified need; convincing evidence of impact; and scope and capacity 
within school to implement the approach), we found instances of the Literacy Octopus interventions 
impacting on teachers’ literacy practices (or other aspects of practice depending on the focus of the 
materials). 


Interviews revealed that impacts on teachers’ practice ranged from small ‘tweaks’ to existing practices 
to more substantial whole-school or key stage reforms of a particular approach. One literacy co- 
ordinator explained the impact of using a new pupil self-assessment marking strategy from the 
Teaching How2s Website (Arm 4) in their school: 


‘This is now incorporated into the whole school marking policy—used by the whole of Key 
Stage 2. | trialled it for a few weeks with the Year 6s, then we evaluated it before changing the 
marking policy and rolling it out. That’s going well—Ofsted and everybody else are keen on 
the children knowing what they need to do, so that’s a good example of one of the ways the 
pupils’ self-assess.’ 


A Year 5 class teacher outlined changes their school had made to group reading as a result of 
attending a ResearchEd conference: 


‘It’s been really good to try something different and reflect upon it. It’s what we were missing 
with group reading; now we’re tackling both sides—the children reading on their own and at 
home and learning to read—which was being neglected. Getting good ideas from the 
conference was the fuse.’ 


Among the case-study and telephone interview schools, perceived impacts appeared to be strongest 
on the individual teachers directly involved in the interventions, though spread more widely to impact 
on colleagues and the wider school where there was some form of collaborative activity (such as 
multiple staff from the school attending a conference, staff meetings and training to disseminate 
information, or whole-school implementation of strategies) and where the senior leadership team had 
supported the implementation of the approaches at a whole-school level. 


Perceived impact on research use 


Interviewed teachers also occasionally identified impacts of engaging with the interventions on their 
awareness, understanding and use of research evidence. Predominantly, this was a general 
awareness and appreciation that the strategies presented were evidence-based. However, several 
teachers described how, as a result of the intervention, they were more aware of the research 
evidence on whether and how a particular approach was effective, or understood more about the 
theory underpinning a particular approach, as one senior leader summarised: 


‘What the newsletter does is the hard work for you, it has helped management decrease their 
workload because it’s already raised the questions and provided links to things that we would 
have had to put the effort in to do. It’s relevant, a lot of the topics are things schools are 
looking at anyway and it gives another viewpoint [and] a much bigger picture ... of your 
profession. | always look at it because | value research. It’s become part of the improvement 
plan; it’s embedded in our school.’ 


In some cases, teachers had been persuaded by the evidence to try out the strategy in their own 
practice and used the research-evidence to convince colleagues to explore the approach. For 
instance, this senior leader who participated in a ResearchEd conference explained, ‘...we were 
convinced that the research shows that even in KS2 the more you're teaching phonics all the way 
through, the better results are’. 


Several case-study teachers described how they would be more likely to participate in future research 
projects or conduct their own action-research because they were convinced of the importance of 
exploring the impacts of different approaches on pupils’ learning and using evidence to inform 
professional discussion and decision-making. Trying out evidence-based strategies helped some 
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schools to gain insights into some of the practicalities of action research and implementing new 
initiatives. This finding may indicate that impacts on teachers’ engagement with research are a result 
of successfully applying research in practice rather than a direct impact of simply being exposed to 
research evidence. One deputy headteacher explained that as a result of participating in a 
ResearchEd conference as part of the Literacy Octopus there was: 


‘...greater awareness of the need to refer to research and the process of making a change in 
the school amongst senior and middle leaders is clearer; you start from an area that needs 
improving and you look at research and come up with some ideas and possibilities, and the 
need to measure and monitor the impact as well. We would like to continue this development 
towards research-based strategies and evaluating our impact and looking at how we can 
improve children’s attainment.’ 


Several case-study teachers described how they were making more use of research-evidence (for 
example, by subscribing to IEE research briefings, using the EEF Teaching and Learning Toolkit 
website, or being part of the ResearchEd twitter community) or intended to do so in the future. 


According to the case studies and telephone interviews, impacts on teachers’ engagement with 
research appeared to be more likely in schools where there was an existing culture of valuing and 
using research evidence to inform practice, where the evidence-base was explicit and from a credible 
source, and where the evidence-based practices were perceived to have been successfully 
implemented (such as the explicit exploration of the evidence on effective peer-reading strategies as 
part of the CEM CPD, or a ResearchEd conference session on how to use and interpret findings 
within the EEF Teaching and Learning Toolkit). 


Although the process evaluation draws on a relatively small number of cases and is therefore only 
suggestive of trends in responses, overall, we found more examples of perceived impact on teachers’ 
action where the Literacy Octopus materials and support ‘transformed’ the research evidence into 
guidance, materials, and approaches that were readily applicable to practice. This finding is 
consistent with wider literature on effective knowledge mobilisation that highlights the importance of 
the ‘transformation’ of evidence for application in practice (Nelson and O’Beirne, 2014; Becheikh et 
al., 2009; Durbin and Nelson, 2014; Sharples, 2013). 


Related to this, we also found examples of teachers changing their practice as a result of the Literacy 
Octopus materials yet without having directly engaged with the research evidence themselves. For 
instance, a teacher had used the How2s website teaching and learning resources without referring to 
any of the additional information about the supporting evidence for the approaches. This finding may 
challenge the notion that using research evidence in practice is necessarily a linear process through 
awareness, understanding, and leading to action. Indeed, some providers predicted in their Theory of 
Change that it would be possible to change teachers’ practice without them necessarily understanding 
the evidence underpinning the approach advocated. Providers noted how teachers were reassured by 
the authority of evidence-based strategies but felt that most teachers would be more concerned with 
understanding the strategy itself and how to apply it than with interrogating the evidence-base. 


Perceived impact on pupils 


Where teachers had implemented the evidence-based practices from the Literacy Octopus materials 
and support, they often reported (or anticipated) positive impacts on their pupils. They reported that 
the new or refined literacy approaches that they had implemented were helping to enhance pupils’ 
literacy skills, confidence, and behaviour. 


Examples of these impacts include: 


e Teachers noted improved collaborative learning, communication, and discussion skills in 
pupils as a result of implementing classroom management and pupil grouping strategies from 
the How2s teaching and learning website. A recently qualified teacher said: ‘it helped me to 
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put routines in place to help the children. [There has been] a huge difference in behaviour ... 
because they have been focused.’ 


e Teachers perceived that their pupils’ vocabulary, reading comprehension, communication 
skills, confidence and motivation had improved as a result of implementing a peer-reading 
scheme and ‘word of the day’ activity from the CEM booklet and CPD sessions. For instance, 
a senior leader in one school commented on the impact of ‘buddy reading’ on children’s 
confidence to read: ‘the children love it, [they] see [it] as non-threatening and something [they] 
can enjoy; it’s really motivational.’ 


e Teachers also perceived enhanced progress for children not meeting expectations as a result 
of quicker and earlier interventions delivered by more effective deployment of teaching 
assistants based on evidence reviewed in research briefing newsletters from IEE. 


e Teachers also felt that their pupils’ spelling, reading, and phonics abilities had improved, as 
well as confidence, as a result of implementing new phonics-based spelling strategies and 
one-to-one phonics tutoring based on ideas and evidence discussed at a ResearchEd 
conference focused on literacy. 


Appendix | provides detailed examples of the impacts of the Literacy Octopus interventions on 
teachers and pupils in four case-study schools. 


Unintended and negative impacts 


There were very few reports of unintended or negative impacts as a result of teachers’ engagement 
with the Literacy Octopus interventions. Occasional comments identified adverse impacts of 
participation on teachers’ time. One provider also gave an example of a teacher who had 
implemented a literacy strategy without replicating the conditions in which it was found in the research 
to be effective. There was no indication of whether this had led to negative impacts, though it 
highlights the potential for this where research evidence is misinterpreted. 


Formative findings 


Two-thirds (60%) of the 120 teachers surveyed said that they would ‘definitely’ or ‘probably’ 
recommend the Literacy Octopus intervention that they had been involved with to another school on 
the basis that it was useful, relevant, included suggestions for implementation of the 
practice/approach, had an impact on practice and pupils, was evidence-based, and was timely. 


One-third of survey respondents would not recommend the Literacy Octopus intervention they were 
assigned to, mainly due to perceived lack of relevance or fit with current priorities, hence initial 
engagement and, indeed, the real-world matching of evidence priorities to schools (rather than the 
random allocation tested in this trial) may be an important consideration. 


Teachers and providers also fed back in interviews how the Literacy Octopus interventions, as well as 
research dissemination approaches more generally, could be improved. Suggestions included: 


e improving the amount and type of communication: for example, more communication and 
more lead-in time for activities and events, more ongoing contact to support implementation of 
approaches, and a more structured timetable of activities to support preparation, action and 
review phases; 


e considering the mode of dissemination: such as more local, regional events and cluster 
working and offering a choice of dates to aid accessibility, digital and hard-copies of materials, 
more face-to-face support (providing opportunities for interaction and discussion and time to 
focus on the topic), optional online activities (to provide a flexible and easily accessible 
alternative to face-to-face activities) such as tutorials, presentations, and discussion forums, 
and easy-to-access collated and thematically organised research evidence and materials; 
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e reviewing the content of materials: for example, more differentiation for ages and abilities, 
more explicit evidence, including theory and evidence of impact of the approaches, 
practitioner experience of the approach, separate issues for primary and secondary phases, 
new strategies/latest thinking/innovative practices, and teacher-led identification of relevant 
research evidence; and 


e improving the dissemination and implementation within schools: for example by more 
practical guidance on how to implement strategies, support for teachers’ own ‘action 
research’, trialling and evaluating approaches and research literacy, encouraging involvement 
of senior leaders, and designated teachers or research advocates in schools to support the 
use of research-evidence in practice. 


For future development, organisations involved in supporting teachers’ research use need to consider 
whether to put efforts into increasing initial engagement (for example, increasing participation by 
greater lead in time, local rather than regional events, and hub-led events), and/or into transformation 
and implementation (for example, through collaborative, planned, and championed approaches). 


Control group activity 


Surveys with control group schools at the beginning and end of the trial period revealed that teachers 
were generally positive about the support they had receive for KS2 literacy from external providers 
during the trial period (a higher proportion of teachers rate the support as ‘very good’ or ‘quite good’, 
than those rating ‘quite poor’ or ‘very poor’—see Table 14). 


The survey also asked teachers in the control condition what kind of support for improving KS2 
literacy teaching and learning they would recommend to other schools. In the order of prevalence, the 
responses were: professional development (such as targeted CPD, peer-to-peer learning, and sharing 
good practice), specific literacy schemes and techniques, and other forms of information and support 
(for example, in-school support, online support, external support, and using evidence). 


Table 14: Control teacher survey—control teachers’ views about the mode of communication 
and dissemination of KS2 literacy support from external providers 


How would you rate the 


following elements of 
the KS2 Literacy 


JU) o) ole) a macter-hce iran (OH) 


Posted materials 


Electronic materials 33 25 31 279 
Websites 15 48 17 19 283 
Evidence fairs or 6 12 12 66 279 
conferences 

CPD/training sessions 31 42 8 18 283 
Online events e.g. 3 42 9 74 279 
webinars 


Teacher 


observation/teacher peer 


Very 
fe fexeye| 


% 


eli) 
good 


Average 


ID )Ce Mae) s 
use 


7) 


Be) c-|| 


igeX<J Led ate liae| 


to item (N) 
281 


learning 


Pupil assessment tools 


15 


282 


A series of single response items. 


A total of 305 teachers completed the control survey; 287 gave at least one response and 18 did not respond to these items. 


Source: NFER Literacy Octopus teacher survey, control, 2016. 
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In interviews with teachers and senior leaders from control schools (six interviews conducted), we 
found that while the schools had signed up to the project because of the interventions on offer, there 
was a good understanding of the purpose of the control group and there was no evidence of resentful 
demoralisation or compensation rivalry. Indeed, in most schools, the staff we interviewed tended to be 
unaware that their school was even involved in a research project if they had not been the initial staff 
member to sign up for the Literacy Octopus. 


Interviews with control group teachers and senior leaders indicated that schools had continued with 
‘business as usual’ in terms of their engagement with research evidence and KS2 literacy practices 
during the trial. With only one exception, the interviewed teachers said that they had carried on with 
the same literacy work and interventions that they had used prior to their engagement with the 
Literacy Octopus project. 


The control group interviews showed varying levels of research engagement. Schools that appeared 
the most research-engaged reported having a research culture within the school that pre-dated the 
introduction of the Literacy Octopus project. Examples of this included having existing research links 
with other organisations, members of staff who had come from a research background (for instance, 
holding a Masters degree in education), and participating in research within school. When looking for 
research evidence to assist in their teaching practice, interviewees cited using the Department for 
Education’s guidance, Google internet searches, social media sites such as Facebook and Twitter, 
and teacher forums such as the Times Educational Supplement (TES). Most teachers agreed that 
research evidence had become more available and accessible, for example through an increased 
volume of advertising materials and phone calls to schools. Some teachers felt that ‘research 
evidence’ was a ‘buzz word’ in education at present. However, several teachers pointed out that they 
were uncertain about the causality of their increased awareness: they were not sure whether research 
evidence really was becoming more available, or whether they were just more aware of it due to other 
factors such as being involved in Literacy Octopus. 
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Conclusion 


Key conclusions 


The project found no evidence that any of the interventions improved pupils’ Key Stage 2 
English scores. The five padlock security rating means we have high confidence in this 
result. 


There was no evidence of impact on any of the six teacher Research Use Measures used 
in this trial. However, we have limited confidence in this result given the low response rate 
to the questionnaires designed to capture these outcomes, and some measures were only 
moderately reliable. 


Schools’ level of engagement varied: six out of ten schools did not engage to the level 
expected by the providers, although a small proportion engaged to a greater extent than 
expected (for example by hosting CPD sessions). Reasons for not engaging included lack 
of time, the timing and location of events, and a preference for face-to-face support rather 
than online or remote formats only. 


Teachers felt research evidence was most effectively communicated when it was 
interactive, accessible, relevant, included a balanced and credible discussion of the 
evidence, and focused on how to apply the evidence in practice. Where schools went on to 
implement changes in light of Literacy Octopus engagement, these came about through 
mechanisms such as in-school collaboration, further enquiry, and trying out, reviewing, 
adapting, and embedding approaches. 


The lack of impact across the different interventions suggests that simply communicating 
research evidence to schools is not enough to improve outcomes. How easily the 
presented evidence can be used in practice—and the conditions in schools for 
implementing evidence-based change—might be just as important. Further research should 
assess whether interventions can transform evidence into practical action, and develop 
supportive implementation conditions in schools. 


Interpretation 


Primary and secondary outcomes 


Literacy Octopus interventions, whether passive or active, made no difference to pupils’ literacy 
attainment at Key Stage 2. There was no evidence that any randomised trial arm contributed towards 
differences in academic progress of pupils in literacy at KS2 (2015/2016). Literacy Octopus 
interventions had no impact on literacy attainment at KS2 for everFSM children. When comparing the 
progress in literacy of everFSM intervention pupils with those in the control group, none of the 
Literacy Octopus arms made a difference. Schools’ level of engagement, which varied greatly, also 
had no impact on differences in pupils’ academic progress in literacy. 


Literacy Octopus interventions made no difference to teachers’ research use as measured on six 
outcomes on a validated research use survey. Measures 1 and 2—‘positive disposition to academic 
research in informing teaching practice’ and ‘use of academic research to inform selection of teaching 
approaches’—were the main secondary outcomes and were analysed using a multilevel model that 
matched baseline and endpoint teacher responses. No statistically significant differences were found 
between either of these measures and the control group. Measures 3-6 (‘perception that academic 
research is not useful in learning’, ‘perception that own school does not encourage use of academic 
research’, ‘active engagement with online evidence platforms’, and ‘your knowledge about research’) 
were analysed using ANOVA on the endpoint teacher survey responses only, and no statistically 
significant differences were found for any of these measures when compared with the control group. 
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Engagement with the interventions and communication approaches 


Schools’ levels of engagement with the Literacy Octopus materials and support was relatively low: 
just over two-fifths of the 540 intervention schools did not engage according to monitoring data; 
providers had expected about one-fifth non-engagement with events. That said, just over one-fifth 
engaged to the level planned for their arm, and around one-eighth engaged to a greater extent than 
expected (for example, they hosted CPD sessions, or they requested further materials). Teachers 
engaged where they felt the materials or support were communicated in an accessible way (such as 
posters that were easy to display, or convenient event locations) and that the content of materials was 
relevant (for example, linked to an existing need). Time, the timing and location of events, relevance, 
the format of support (some teachers would have preferred some face-to-face support while others 
valued the flexibility of online activities), and the staff member’s capacity and authority to implement 
strategies and learning from the materials were barriers to engagement (the latter indicating the 
importance of senior and middle leadership buy-in, which is supported by existing literature (Speight 
et al., 2016 and Nelson and O’Beirne, 2014). 


Teachers who would not recommend the Literacy Octopus intervention to others highlighted a lack of 
relevance and other priorities, which suggests the importance of matching research-based material 
and support to teachers’ needs (this was not fully possible on this trial as schools were randomly 
allocated). This reflects recent research which reveals the importance of supporting teachers’ own 
evidence-based queries and matching them to evidence-based support (Lord et al., 2017), and helps 
to reinforce the importance of teacher-led problem-solving described in Becheikh et al.’s (2009) model 
of Knowledge mobilisation. 


All Literacy Octopus providers delivered mainly as planned (fidelity), but made two main changes: (1) 
additional invitation approaches (for example, additional phone calls to schools) were made where 
initial levels of response to invitations was low, and (2) alternative active support, for example 
webinars, was arranged for teachers who had wanted, but were unable, to attend an event. This 
shows that ‘push’ dissemination approaches required extra effort from researchers in order to 
facilitate initial engagement from teachers. 


Further implementation and perceived impact 


The preference in the qualitative findings for more supported activity (such as face-to-face contact or 
guidance on practical application) bears out in the quantitative evidence which generally shows more 
positive means for each provider's active arm compared with their passive counterpart across the 
Research Use outcomes. However, the passive/active nature of support did not appear to be the key 
determinant as to whether schools engaged and took further action—there were both high and low 
levels of engagement in both active and passive arms. 


Where schools went on to implement changes in light of Literacy Octopus engagement, these came 
about through mechanisms such as in-school collaboration, schools undertaking further enquiry or 
follow-up activity with the provider, schools trying out, reviewing, adapting, and embedding 
approaches, and having a planned approach to implementation. These more_ interactive 
implementation approaches reflect the existing evidence that more socially interactive and 
collaborative forms of research engagement are more effective (Becheikh ef al., 2009; Nelson and 
O’Beirne, 2014; Sharples, 2013). Where teachers did engage, most perceived that participating had 
led to positive impacts on their practice, enabling them to reflect on, reinforce, and change their 
classroom practice, and discuss best practice with colleagues in their school. 


Another factor in successful implementation, according to the case studies in this evaluation, was 
support from a within-school research champion or research lead who could cascade information, 
facilitate staff meetings, and, where they had authority, drive changes in the school. This reflects other 
recent research that highlights the importance of within-school research champions in driving 
research use in schools (Griggs ef a/., 2016; Lord et a/., 2017). 
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Further development 


Literacy Octopus interventions were all reasonably light touch, and as, overall, they had no impact on 
the primary and secondary outcomes measured in this trial, providers of research evidence need to 
consider the findings from the IPE of this study that highlight the need to focus on transformation and 
implementation factors. This recommendation fits with the findings from the sister ‘passive trial’ 
showing that simply disseminating research evidence to large numbers of schools does not impact on 
attainment. The trial also revealed that many control group schools were receiving support for KS2 
literacy as part of business as usual, and in interviews some spoke about using research evidence to 
inform decisions. Like their intervention counterparts, control group schools would like more 
collaborative CPD opportunities to improve literacy strategies. 


Providers of research evidence therefore need to consider where to focus their efforts. Two areas 
have been highlighted here: first, the need to increase initial engagement (for example, through 
greater lead-in time, hub-led or local rather than regional events, and more interactive communication 
approaches), and second, by supporting implementation (for example, with translated, applicable, and 
collaborative approaches). The former may reach more schools—and in a reasonable, affordable way 
(all interventions were very low cost in terms of cost per pupil per school); the latter might be more 
expensive to provide and hence, from provider-costs point of view, might need to focus on fewer 
schools. 


Limitations 


The primary outcome analysis for this study had almost no limitations in terms of bias or precision as 
it used administrative NPD data; there were only minor incidences of missing data where four schools 
(1%—three control and one intervention school) could not be found on the NPD supplied data purely 
for administrative reasons such as academisation, amalgamation, or school URN changes. Pupil-level 
missing data, however, through individuals not having a full complement of test scores at baseline 
and follow-up, accounted for 6%. Our exploration of any imbalance at baseline suggested that this did 
not impinge on results. Furthermore, it is unlikely that Literacy Octopus interventions interfered with 
pupils’ tendency to complete a KS2 test. 


The trial was powered to detect an MDES of 0.121; it may be that any changes in attainment were too 
small to detect or that insufficient time had elapsed to allow any changes in practice to become 
embedded in teaching and learning. Results for the 2016/2017 cohort, to be published in 2018, will 
investigate this second aspect further. That said, the sister passive trial which was designed with an 
MDES of 0.024 did not find any evidence of impact on pupils’ literacy attainment of large-scale ‘push’ 
communication approaches to sharing research-evidence with schools. 


The secondary outcomes analysis for this study was based on one-time-point school-level measures 
(Measures 3, 4, 5 and 6) and (for Measures 1 and 2) two-time-point teacher-level responses to the 
Research Use Survey (Poet ef al., 2015, unpublished). For one-time-point measures, results of the 
ANOVAs showed no effects for Measures 3, 4 and 5. The ANOVA for Measure 6 was statistically 
significant but when performing the post-hoc Dunnett's intervention versus control comparisons, no 
effects were seen. The completers’ model used for the two-time-point measures had a reasonably low 
number of respondents (335). When further ‘outcome survey’ respondents’ responses were used in 
multiple imputation, the results attenuated, suggesting bias in the completers sample (for example, 
they were perhaps more likely to be research-engaged, or were more closely involved with the 
Literacy Octopus). All survey measure results deserve this caveat: just over half of intervention survey 
respondents had had some direct engagement with the Literacy Octopus (themselves or cascaded 
via a colleague), hence there was a large minority of respondents who had not had direct 
engagement and whose responses were either a reflection of ‘school wide reach’ or indirect 
engagement, or they were unaware of any engagement. 
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While findings from the primary outcome are generalizable (that is, disseminating evidence-based 
materials does not in itself have an impact on pupil attainment), the two significant findings from the 
Research Use outcomes models (LRT for Measure 2 and ANOVA for Measure 6) need to be 
interpreted with caution as they are based on lower response rates than planned and seem to be 
related to between-intervention arm differences. The Dunnett’s Tests showed no differences between 
each intervention and control. 


Engagement levels in the Literacy Octopus were lower than expected (for example, providers 
anticipated around one-fifth non-participation in events and all providers had to put in place additional 
initial contacting in order to encourage participation). All results in the study, therefore, relate to lower 
levels of engagement than planned. Comments from both intervention and control schools suggest 
that this reflects the pressured, real world market place in which schools operate—in that they do not 
take up every intervention or CPD opportunity that lands through their door—and was not a reflection 
on the Literacy Octopus itself. That said, the CACE analysis, which explored schools’ level of 
engagement, found no association between level of engagement and pupils’ literacy attainment (the 
primary outcome). 


Schools’ low engagement and the limited effect on Research Use (according to Dunnett’s Tests, 
which showed no effect on Research Use on any of the measures in any of the arms, compared with 
the control group) were borne out in the case-study sampling that aimed to explore examples of 
changes in practice and mechanisms for change. Even where engagement was high (according to Ml 
data and interim survey responses), finding case studies where schools had gone on to implement 
KS2 literacy evidence-based strategies or engage with research evidence, proved challenging—it was 
a minority of schools that had progressed to implementing or using research evidence. Hence, all IPE 
data in this study needs to be interpreted with a minority of schools in mind, and is not generalizable 
to the whole school population. The case studies and detailed impact trails from those schools help us 
to understand better the mechanisms that support research engagement and implementation (such 
as leadership support, in-school collaboration, trying out and reviewing, and so on). 


Finally, a noteworthy limitation in this multi-armed trial was that of multiple treatments. Although this 
design clearly brought benefits in recruiting and analysing all under one roof, for some schools, there 
was a perceived lack of fit in what they were allocated to. Their engagement in the dissemination and 
support activities may therefore have been compromised as compared to what might have been 
possible in a series of parallel group trials. 


Future research and publications 


The impact on pupils’ literacy attainment in Key Stage 2 for the previous and successive cohorts 
(2014/2015 and 2016/2017) will be analysed and reported in an addendum report in 2018. This 
reflects the Theory of Change that changes in pupil attainment may take time to surface, and hence 
we explore three cohorts’ worth of attainment data. (Note, because of the 2015/2016 nil results, and 
because research evidence suggests that it takes time for pupil attainment impacts to surface through 
teacher-change activity, it is unlikely that we would find any impact on the 2014/15 cohort. We will 
discuss whether it is appropriate to conduct this analysis for the addendum report.) 


The passive and active dissemination strategies used in the Literacy Octopus did not have an impact 
on pupil attainment and did not make a difference, when compared with the control group, to research 
use. However, the IPE found case-study examples of changes in awareness, understanding, and 
practice. Further quantitative and qualitative research is needed to explore whether further 
transformation and implementation factors, over and above dissemination approaches, impact on 
schools’ or teachers’ research use and pupil attainment. In particular, what other ingredients are 
needed in research-dissemination strategies that might work for the wider population, and how to 
support the development of the conditions for research engagement in schools. 
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Appendix A: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found here. 
Cost ratings are awarded as follows: 


Cost rating Description 


EEELEL Very low: less than £80 per pupil per year. 
Low: up to about £200 per pupil per year 


Moderate: up to about £700 per pupil per year 
High: up to £1,200 per pupil per year. 


Very high: over £1,200 per pupil per year. 
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Appendix B: Security classification of trial findings 


ns : Initial 
Criteria for rating 
Criteria for ratin score 


Fair and clear  quasi- 
experimental design for 


comparison (e.g. RDD) with} wpES< 7 Adjustment 
appropriate analysis, or 03 11-20% for Balance 
experimental design with [0] 


minor concerns about validity 


38 Well-matched comparison 
(using propensity score 
matching, or similar) or] wpES~< 


experimental design with 0.4 21-30% 

moderate concerns about 

ee Adjustment |__| 
- Weakly matched comparison eae 

or experimental design with | MDES < : ee 

; 05 31-40% validity 
major flaws ; 
= te eee 

18 : ‘ 

Comparison group with poor 

or no matching § (E.g. MDES < 41-50% 

volunteer versus others) 0.6 
OR 

No comparator Uae > | over 50% 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks 
e Reason for adjustment for balance (if made): N/A 

e Reason for adjustment for threats to validity (if made): N/A 

e Final padlock score: initial score adjusted for balance and internal validity = 5 padlocks 


The design is a randomised controlled trial. MDES is 0.12 at randomisation, and pupil level attrition is 
approximately 6% for the primary analyses. No threats to validity or imbalance are present. 


19 Attrition should be measured at the pupil level (even for clustered trials) and from the point of randomisation to 
the point of analysis. 
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Appendix C: Information sheets 


Appendix A includes copies of the Literacy Octopus information sheet for schools, the invitation letter 
used in recruitment to the trial, and the reply and consent form that schools completed when signing 
up to the trial. 
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The Literacy Octopus: Improving Literacy at 
Key Stage 2 


What are the purposes and aims of the project? 


A major new initiative has been launched to help teachers to improve pupils’ Key Stage 2 
literacy, all based on the latest research evidence. The project is evaluating how best to support 
primary schools’ literacy improvement. 


Who is sponsoring this work? 


Funded by the Education Endowment Foundation (EEF), Department for Education and the 
London Schools Excellence Fund, this initiative involves a number of high profile organisations 
helping schools to improve literacy in Key Stage 2 through evidence-based support and 
materials. International evidence shows a strong link between high performing education 
systems and the effective use of research evidence in schools. The project will explore different 
ways of engaging teachers and school leaders with best practice approaches and measure their 
effect on pupils’ attainment. 


What activities and resources are on offer? 


Schools that take part will be randomly allocated to receive free support or materials via one of 
several different approaches, all designed to improve Key Stage 2 literacy by drawing on 
research evidence. Activities range from light touch engagement with materials, to more active 
support through CPD sessions and online help. Some schools will be invited to attend seminars 
and conferences about literacy and intervention programmes, and/or receive additional training 
on literacy strategies. Other schools will receive improvement materials through the post or 
online; this may include regular mailings of new reports about teaching and literacy, or access to 
interactive websites about teaching techniques. 


What does taking part involve? 


To participate in the project, a small number of school staff need to complete a ‘baseline’ survey 
during the autumn term 2014. Participating schools will then be randomly allocated to one of nine 
support groups, or a control group. Each group will receive a different type of evidence-based 
literacy support and/or materials, during the spring and early summer term of 2015. Staff will 
need to complete a follow-up survey in spring 2016. 


How will schools benefit from taking part? 


This is an exciting opportunity for schools to receive free support from high profile education 
organisations, to support school improvement. We are trying to find out the very best ways of 
supporting schools to improve literacy through evidence-based resources. So whichever activity 
you are involved with will be a highly valuable part of the project, and will enable your school to 
receive improvement support. Note in some cases, schools will need to organise and pay for 
supply cover for staff to attend training or events which will be taking place in London and 
Leeds/York. 


If your school is part of the control group, this is also extremely valuable to the trial, and you can 
have access to many of the materials and resources after the trial has ended. 
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What will schools receive for participating in the project? 


We will offer each teacher who takes part in the final survey in Spring 2016 a £5 ‘thank you’ in 
the form of an e-Amazon voucher or a donation to charity. Each school in the control group will 
be sent a £30 book token for their school, in anticipation of their commitment to participate in the 
final survey. All schools will receive feedback on the results of the surveys after the final survey 
has been completed. 


Do schools have to take part? 


No, this is voluntary but primary schools (and those middle schools with Year 5/6 classes) in the 
North and South of England will be invited to take part. There will be 780 schools involved in the 
trial. 


How will NFER use and protect the data collected? 


All questionnaire responses will be treated with the strictest confidence and will be used for the 
purposes of this evaluation only. The study also involves NFER analysing end of Key Stage 2 
national curriculum assessments in conjunction with baseline Key Stage 1 results, to see if there 
are any links between research engagement and pupils’ performance in literacy. The enclosed 
reply form asks for your school’s permission for NFER to access data about your pupils, using 
Pupil Matching Reference (PMR) numbers, from the National Pupil Database. 


The data collected from the questionnaires will be matched to the National Pupil Database and 
will be passed onto EEF and the Fischer Family Trust (who manage EEF’s data archive), and 
stored on the EEF data archive and UK Data Archive for research purposes. Whilst the EEF and 
FFT data archives will contain NPD matched data, no individual school, teacher or pupil will be 
identified in any report arising from this evaluation. The data kept by the UK Data Archive will be 
anonymised. 


How will the findings be used? 


The findings from the project will be freely available on the EEF’s website. They will be used to 
inform the education sector about the most effective ways of engaging schools with research 
findings relating to literacy, to support schools’ continuous improvement and development. 


Which other organisations are involved? 


Six partner organisations will deliver the support to schools: the Institute for Effective Education 
(at University of York); Campaign for Learning in partnership with Train Visual; the Centre for 
Evaluation and Monitoring (part of Durham University); and NatCen with ResearchEd. 
www.nfer.ac.uk/octopus provides further details about the organisations’ activities. The NFER 
will carry out the independent evaluation. The NFER is not involved in the direct delivery of the 
support activities or materials. 


What is a randomised controlled trial (RCT)? 


Schools that sign up to a Randomised Controlled Trial (RCT) are allocated to a group ina 
random way (the equivalent of flipping a coin). This means that any characteristics that schools 
have should be evenly spread across the different groups taking part in the trial. Random 
allocation is essential to the evaluation as it shows that any difference seen in ‘intervention’ or 
‘treatment’ schools is because of the intervention and not any other factors. Therefore, an RCT 
can be the best way of outlining what effect an intervention has on children’s attainment. (Please 
see this link for more details on RCTs: http://www.nfer.ac.uk/research/trials-unit/guide-to- 
trials.cfm) 
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RPO/EEFA/41195/2 


A Jucatior dos SUPPORTED BY 
Endowment | Department MAYOR OF LONDON 
The Headteacher | —- 
«Description» 
«Address» 
«Address2» NFER No: 
«Address3» 
«TOwn» 
«County» 


«Post_Code» 
6" October 2014 


Dear Headteacher 


Take part in activities to improve Key Stage 2 literacy 


You are invited to take part in a major new initiative to help teachers to improve pupils’ Key 
Stage 2 literacy, all based on the latest research evidence. The initiative is funded by the 
Education Endowment Foundation (EEF), Department for Education and the London 
Schools Excellence Fund, and is exploring a range of best practice approaches to 
supporting schools’ literacy development and their impact on pupil attainment. Your school 
has been sampled for this project (known as the Literacy Octopus trial) and | am writing to 
invite your participation. This is a nationwide initiative involving a number of high profile 
education organisations offering literacy resources and/or support to schools. NFER has 
been funded to carry out the evaluation of the project, and we very much hope that taking 
part will support your school’s continuous improvement and development. 


Activities and resources on offer 


Schools that agree to participate in the evaluation will be randomly selected to receive free 
support or materials via one of several different approaches, all designed to improve Key 
Stage 2 literacy by drawing on research evidence. Some schools will be invited to attend 
seminars and conferences about literacy and intervention programmes, and/or receive 
additional training on literacy strategies. Other schools will receive educational materials 
through the post or online, including regular mailings of new reports about teaching literacy, 
or access to interactive websites about teaching techniques. Some schools will be allocated 
to be part of a control group and will be asked to complete the staff surveys only. They will 
not receive information or participate in activities as part of the trial, but will have access to 
many of the materials and resources after the trial has ended. More information about the 
support being provided by the organisations involved is given on the enclosed information 
sheet. 


What will it involve for your school? 


To participate in this project, we would be grateful if you, your Key Stage 2 literacy 
coordinator, other Key Stage 2 teachers and leaders would complete a brief 15 minute 
online survey that asks for views about the different influences on decisions you make in 
your teaching and school policies. Questionnaire responses are needed by the 28" 
November 2014 from you and your staff to take part in the trial. Survey access details are 
given overleaf. 
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We also ask that you complete and return the enclosed reply form to provide your consent 
for your school to be involved in the evaluation and for NFER to access anonymised 
National Pupil Data (details about the purpose of accessing the pupil data are given on the 
enclosed information sheet). Schools will be contacted in early January 2015 with 
information about next steps. Resources will be sent to schools and activities will take place 
in Spring and Summer 2015. 


The final activity we would require is that staff at your school, such as the Key Stage 2 
literacy coordinator and literacy teachers complete a follow-up survey during Spring 2016 
giving their views. All individual teachers that take part in this final survey will be a given a £5 
Amazon voucher as a token of our appreciation. Schools that are randomly selected to be in 
the control group will also receive a £30 book token alongside access to many of the 
materials and resources. In addition, all schools will be provided with feedback on the results 
of the surveys after the final survey has been completed. 


How the data will be used 


Please note that the name of your school and your staff contact details will be provided by 
NFER to one of the four delivery partners (see information sheet). This information will be 
used solely for the purpose of enabling the research partners to contact your staff with 
information regarding this research. In addition, all of the data collected by NFER and its 
research partners in this evaluation, including data from the National Pupil Database 
matched to named data from pupils in your school, will be provided to Fischer Family Trust 
(FFT, which is the organisation appointed to manage EEF’s data archive) and stored in the 
EEF data archive and the UK Data Archive for research purposes. The overall findings will 
be included in a publicly available report used to influence practice nationally. Please note 
we will not use pupil names, staff names or the name of your school in any reports arising 
from the research. 


To take part in this evaluation 


Please complete the enclosed reply form as soon as possible and return it to the NFER in 
the pre-paid envelope provided. In addition, please complete the online survey and also 
pass on the website and login details (provided below) to a selection of your staff members. 


We would be most grateful for your help with this research and hope you will be able to take 
part. In the case of local authority maintained schools, we have notified your local authority 
that we would be contacting you. | look forward to receiving the reply form and online survey 
responses from your school so that you can take part in the trial. If you have any queries 
about this evaluation, please do not hesitate to contact me by email to octopus@nfer.ac.uk 
or telephone on 01753 637218 or my colleague Dave Hereward on 01753 637352. 


Yours sincerely 


XXX 


Research Manager, Research and Product Operations 
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Instructions for accessing the online questionnaire: 


School staff can access the questionnaire by visiting the following website address: 
www.nfer.ac.uk/EEFA/survey 


Once at the above website, each member of staff should then enter this survey password number to log in to 
the survey: «NFER_No». Instructions for completing the questionnaire will then appear online. The 
questionnaires should take approximately 15 minutes to complete. We would be most grateful if up to around 
five members of staff would take part in the survey by 28" November, including the Headteacher, your Key 
Stage 2 literacy coordinator and other Key Stage 2 teachers and leaders. 
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@D National Foundation for Educational Research 
A Bow SUPPORTED BY Research Operations, The Mere, Upton Park, 
idence tor ndowm Department MAYOR OF LONDON Slough, Berkshire, SL1 2DQ 
— eae Nea Telephone 01753 637007 
Fax: 01753 790114 
Email address: octopus@nfer.ac.uk 
RPO/EEFA/41195/2a NFER No: «NFER_No» 
Reply form 
Take part in activities to improve Key Stage 2 literacy 
Are your details correct? Please Amend NFER 
use 
School Name «Description» 
Headteacher «Head_ Title» «Head_Forename» 
«Head_Surname» 
Tel. No: «Phone» 
Fax No: «Fax» 
Email: «Email» 


Please read the following statement and then sign below to provide your consent for your school to participate in 
the evaluation: 


| understand that the name of my school and the staff contact details provided on this reply form, and by my staff in 


the online staff questionnaires, will be provided by NFER to its research partners. | also understand that all of the 


data collected by NFER and its research partners during the course of this evaluation, including National Pupil Data 
matched to the school, will be provided to the Fischer Family Trust (FFT). All of the data provided to the FFT will be 
stored in the Education Endowment Foundation’s data archive and also entered into the UK data archive in 
anonymised format for additional future analysis, to be used for research purposes. 


| have read and understood the statements given above regarding the use of the data collected by NFER and its 
research partners during this evaluation and | agree for my school to take part in the evaluation and for the NFER to 
access pupil data from the National Pupil Database. 


Headteacher/SMT Signature: ............cccccccceccsssessesssescsssesesssessscssscscsesseseseseeees Date? i iediscasteiedeiedesieeseattasiecaasics 
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Please provide the name of a member of staff at your school with whom NFER can liaise with regarding this evaluation: 


Name of contact in the school: Mr / Mrs / Miss / Ms / Dr 


Contact phone NUMBE?: 1.0.0.0... ce cescesesceeceeeeeceeeeeseseeeseeceeeneaes Contact job title: 0.0... cece cee ceeteeeceeceeees 


Contact: ermallll Ad MOSS! iciiciics cccsssecacssaceccusecicaseaveanincsusaasscedsaisnreagcascedantunecessucdaciedesiaddesenssa esndeeicausedéenlscadeansaccueasbeceea sdnctused 


If you cannot help us on this occasion, we would be grateful if you could let us know the 
reasons why your school cannot take part. This will help us understand the factors schools 
consider when deciding whether or not to participate. 


Please return your completed reply form to NFER by fax on 01753 790114 
or by using the addressed pre-paid envelope provided. 


Thank you very much for your help. 


If you have any queries regarding this research please do not hesitate to contact 
XXX at NFER on octopus@nfer.ac.uk or 01753 637218. 
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Appendix D: Literacy Octopus providers’ Theories of 
Change 


At the start of the trial, the four Literacy Octopus intervention providers were asked to complete a 
Theory of Change diagram in collaboration with NFER. These were reviewed with providers during 
interviews in 2015 and again in 2016. A Theory or Change depicts the process by which planned 
intervention strategies are anticipated to lead to changes in outcomes for the recipients. This 
appendix presents each of the Literacy Octopus providers’ Theories of Change using the project 
template as well as some providers’ existing logic models for their interventions”. 


Comparing and contrasting across the interventions 
In all four Theories of Change the ultimate planned impact is to influence teachers practice and 
behaviour though the use of research-evidence. 


IEE and ResearchEd predicted hierarchical impacts: first on teachers’ awareness then understanding 
of the research evidence, then action. The Teaching How2s ToC also predicted similar hierarchical 
impacts (awareness, understanding, action); however these were related to the How2s approach (i.e. 
visual teaching and learning techniques and teacher-led peer-to-peer support) rather than the 
evidence explicitly. l.e. they suggested it was not necessary for teachers to understand the research 
evidence underpinning a particular approach in order to change their practice. Likewise, CEM’s ToC 
predicts that teachers’ action can change without necessarily first altering their awareness and 
understanding (for example, where materials provide top tips for action underpinned by, but not 
explicitly directive about, the evidence). However, all four providers predict that there will be a 
distinction in the impacts between the active and passive arms of their interventions and that the 
active interventions would be more likely to lead to change, in particular to impact on teachers’ action. 


CEM and ResearchEd were focused on creating an impact on the classroom teacher. Whereas, IEE 
and Teaching How2s sought to impact at a whole-school level by focusing on headteachers and 
senior leaders, which for How2s would then be cascaded to the whole staff. 


In follow-up discussions with providers they reflected on their original Theories of Change. The 
providers indicated that where there was a clear distinction between active and passive interventions 
and where this meant a higher level of social interaction with teachers (e.g. training, conferences, 
ongoing online support) they felt the hypothesis still held that the more active interventions would lead 
to higher levels of engagement and impact. In reality, for some of the interventions there was little 
distinction in how schools engaged with the active and passive interventions (e.g. ResearchEd Arms 
8 and 9) and therefore it was difficult to evaluate this hypothesis. 


The providers noted that school senior leaders’ and teachers’ prior interest in, and engagement with, 
research evidence were important factors mediating the extent of engagement and impact of their 
interventions. 


Providers also reflected on the process of impacting on teachers’ practice. CEM slightly modified their 
original stance and suggested that it was important for teachers to understand the underpinning 
theory and research on an approach in order to change their practice (their booklet particularly 
supports this approach by setting out the evidence for certain strategies; and in the twilight CPD 
session we observed the CEM researcher briefly talk through a recent research paper on cognitive 
reading strategies). 


20 Teaching How2s did not complete a ToC document using the project template. 
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Draft Theory of Change for Literacy Octopus Trial (IEE Arms 1 and 2) 
PASSIVE ACTIVE MEDIATING OUTCOMES "FINAL OUTCOME 


: | } 


MEDIATING OUTCOMES 


Awareness Understanding Assigned Independent Increased 


= Action Action Literacy 


Attainment 


Tracking data 

NFER Questionnaire 

Further engagement with research 
Conference attendance 


Writing a Blog 
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IEE (Arm 2): A Theory of Change 
(ToC) for Research Evidence for 
Literacy Excellence (REFLEX) - 
active 


Overall purpose 


To investigate whether evidence 
fairs and related activities can 
improve teaching and pupils’ 
attainment. 


Purpose for your 
intervention(s) 


Encourage schools to use research 
evidence to guide their decision 
making, by bringing key staff from 
several schools together at evidence 
fairs. Using their own data and 
priorities, and by working through 
some sample scenarios, encourage 
them to select appropriate 
interventions. Connect them with 
providers, and peers who have used 
the interventions, to support 
successful implementation. 


Assumptions 


Headteachers are open to new 
ideas and want their schools to 
improve. 


Ownership of the process by 
senior leadership is essential. 


Research evidence must be 
supported by face-to-face 
brokerage to enable schools to 
select the most appropriate 
evidence-based interventions for 
their situations. 


Improvement in attainment is 
most likely to occur if specific 
interventions are adopted. 


Impact 


Outputs: School leadership engage with the 
scenarios to refine their needs for raising KS2 
literacy. Schools select interventions that meet 
these needs. 


Outcomes: Schools implement the interventions, 
also raising the skills of staff. Improvement in pupil 
attainment. 


Impact: Increased awareness of research 
evidence. 


Wider impact: Schoo! staff will become more 
accepting of research-based interventions, and 
supportive of their use. Schools will become more 
discerning in their selection of interventions, and 
more resistant to the pendulum swing of fashions 
in education. 


Strategies 


What is your approach? The evidence fairs will 
bring schools together to use scenarios and their 
own data to identify effective 
strategies/interventions. Heads and KS2 literacy 
co-ordinators from each school will be invited. 


What strategies and tools will you use? 
Intervention providers and schools that have used 
the interventions will provide support and advice. 
Before and after the evidence fair, continuing 
support will be provided to ensure that solutions 
meet school needs and are implemented 
effectively. 


What resources will you need? This will be 
supported by a range of communications, including 
regular emails, mailings, and telephone calls. 
Magazine (one every term), e-newsletter (one 
every fortnight) and website will be sent for 
calandar vear 2015 


Target Groups 


The main target group are the heads/SLT and KS2 
literacy co-ordinators of the schools. 


KS2 literacy teachers (and TAs who support them) will 
be indirectly targeted/influenced. 


Governors and parents will also be indirectly influenced. 


Teaching How2s: Theory diagram for Arms 3 and 4 


Knowing what Knowing how 


Verbal Tell me what Tell me how 


WAU FI Show me what Show me how 


Procedural knowledge needs to be 
communicated effectively. 


The Teaching How2s approach is based on carefully designed, step-by-step visual guides that walk teachers through 
activities that they can use in their classroom. The theory is that visual guides can accurately communicate evidence-based 
teaching strategies to teachers and help implement them in practice. In addition, where schools use the tool as part of peer-to- 
peer teacher learning, greater impacts on teachers’ understanding of the strategies are likely to occur. Moreover, where 
schools embed the How2s approach into CPD in the school, greater impacts on teachers’ practice and pupils’ learning are felt 
to be more likely to occur. 


CEM ToC (Arms 5, 6 and 7): 
‘Embedding research-based 
interventions for improving Impact 
literacy into teachers’ 


a Outputs: Teachers engage with trial, evidenced through access to pdf/use of hardcopy, attendance 
everyday practice’. 


at CPD sessions, number of completed peer-to-peer observations and pupil diagnostic assessments 


(InCAS). 
O—————— aT 


Overall purpose 


To investigate different ways to 
help schools use evidence to 
improve teaching and pupils’ 
attainment. 


| 


Purpose for your 


intervention(s) 
Whilst research has suggested 
that it takes many hours of 
training to change professional 
practice, it is not clear how to 
effectively encourage and 
engage teachers to utilise 
evidence-based research 
advice, or how much more 
benefit can be gained from 
providing different types and 
levels of engagement 
resources/activities. This 
research aims to establish the 
impact and cost effectiveness 
of a series of increasingly 
intensive methods for 
embedding research-based 
advice into teachers’ everyday 
practice on a large scale, to 
improve end of Key Stage 2 


literacy. 


Teachers can be encouraged 
to engage with research 
evidence and incorporate it 
into their classroom practices. 
Tymms and Merrell (2006) 
reported that the provision of 
research-based advice in the 
form of a simple booklet with 
strategies to help severely 
inattentive, hyperactive and 
impulsive young children was 
effective at improving 
behavioural and attainment 
outcomes among all young 
children, not only those with 
behavioural difficulties; the 
more frequently —_ teachers 
reported using the booklet, the 
higher the pupils’ attainment. 
Other researchers have 
reported other methods to also 
be effective, including 
attendance at Continuing 
Professional Development 
(CPD) sessions, peer 
observations and __ in-class 
coaching (Joyce and Showers 
1980; Sparks 1986; Adey et al. 
2004). This study — will 
incorporate a range of levels of 
intensity of these approaches. 


Assumptions 


Outcomes: Teachers incorporate the evidence-based strategies from the booklet into their everyday 
practice with the aim of improving class literacy. Teachers’ awareness (of research use) of action 
(implementing research based strategies) may be more apparent during the trial for teachers in Level 
2 & 3 who will actively reflect upon their practice and use of the booklet via CPD session(s) and 
peer-to-peer observations (for Level 3 only). At Level 1, we would expect most teachers to acquire 
new knowledge from the booklet, and engaged with its content by implementing suggested 
strategies in their classroom; Level 1 teachers however may be unaware that they are implementing 
research-based evidence as they do not have the opportunity and are not encouraged (by the 
intervention) to reflect upon their practice (i.e. from attending in-service CPD sessions or 
participating in peer-to-peer observations). This can be evidenced by assessing how frequently 
teachers use the booklet and implement strategies within the classroom. At Level 2, we would expect 
teachers to engage and implement strategies from the booklet more frequently and to understand 
that they are engaging with research-based evidence more than Level 1 teachers. This is because 
Level 2 teachers will have had the opportunity to reflect upon and discuss the booklet by attending 
one in-service CPD session, this will help to foster engagement with and understanding of the 
booklet’s content and purpose. We would expect to see the most engagement with the booklet and 
implementation of its strategies within the classroom among Level 3 teachers, as they will have had 
more opportunities to actively reflect upon and discuss the booklets purpose and assess its 
usefulness (by attending two in-service CPD session, participating in two peer-to-peer observations 
and acquiring pupil assessment data (InNCAS)). With a greater understanding of the research-based 
evidence, and engagement with the booklet, we would expect behavioural change among Level 3 
teachers. This can be evidenced via the frequency in which they implement strategies from the 
booklet in their classroom, and by gathering data as to whether they have used the booklet’s 
strategies consistently when teaching both their year 5 and 6 cohorts. Throughout all Levels, we 
would like to know which booklet format (i.e. hardcopy or pdf version) teachers engaged with the 
most. 


Impact: Improved literacy attainment at the end of Key Stage 2. Further contact from 
schools/teachers, particularly post-intervention in the active arms, is an indication of impact/interest 


generated directly through the evidence-based materials and sessions provided. We will log all 
contact and requests we receive from schools as evidence of impact outcomes. 


Strategies 


What is your approach? Level 1 will receive evidenced-based materials only (booklet via post and PDF 
with recommendation for use by Y5/6 teachers, ‘Tips for Teachers’ card and monthly posters to foster 
engagement with the research). Level 2 will receive the same evidence-based materials as Level 1 and 
one CPD session. Level 3 will receive the evidenced-based materials, two CPD sessions, participate in 
two peer-to-peer observations and receive diagnostic information from pupils’ assessments (INCAS) to 
help them tailor the appropriate strategies to their pupils’ needs. All levels will receive monthly contact 
(one per month, via email or telephone contact) from the research team. 


What strategies and tools will you use? Booklets which briefly explain how children learn to read and 
evidence-based strategies on how to improve reading and spelling. ‘Tips for Teachers’ cards will 
summarise the main ideas of the booklet and can be placed in an obvious place as a prompt for 
teachers. Poster, to be displayed in the classroom, will encourage pupil engagement as well as teacher 
engagement with the research. In-service CPD sessions will explain the information in the booklet, share 
good practice and, for Level 3 only, explain how to use the diagnostic information from InCAS. In-class 
peer-to-peer observations will facilitate discussion between teaching peers of the evidence-based 
teaching strategies. INCAS will provide Level 3 teachers with information surrounding their pupils 
progress, pinpoint areas of weakness and consequently teacher target teaching strategies from the 
booklet to improve literacy outcomes. 


What resources will you need? Time, research staff, development and production of materials, CPD 
session locations (participating schools), ICT. 


Which wider partners can help? How? EEF; NFER who are responsible for the recruitment of schools 
and overall evaluation; venues to host in-service CPD sessions. 


Target Groups 


Key Stage 2 classroom teachers who are interested in engaging with and utilising research-based 
evidence to improve their literacy teaching strategies. 


ResearchEd (Arms 8 and 9): A Theory 
of Change (ToC) for ResearchEd 


Overall purpose 


To investigate different ways to help teachers and 
schools to use evidence to improve teaching and 


pupils’ attainment in relation to Literacy. 


g 


Purpose for your intervention(s) 


Increase how informed and engaged 
teachers are in relation to evidence 
based issues 


Wider use of research evidence by 
teachers post intervention 


Help them to locate evidence and 
discern between good and poor 
evidence as basis for teaching 
strategies 


Engaged, informed teachers Improve 
performance and tailored support to 


pupils 


Using Research Ed conference to 
communicate and engage with issues (Arm 
8 and 9) 


Online pre-post event peer learning 
sessions to engage, strengthen and embed 
learning into practice(Arm 9) 


Access to the conference resources post 


Assumptions 


For your approach to be effective, what 
background assumptions have to be true? 
What evidence do you already have that the 
stages of mechanism will be effective, both 
from your own work and others? 


e Teachers will attend conference and 


have means/ability to engage with online 


events 


ResearchEd is already well attended, 
well thought of, and seems (although no 
evidence) to be leading to change 


L&D events suggest online events 
increase learning and engagement and 
help embedding into practice — 
overcoming hurdles collaboratively and 
sharing what works. 


Impact 


Outputs: 


e Teachers attend and engage 
(conference and online events) 


e Learning is shared with peers in 
school 
Outcomes: 


Practice becomes more evidence 
informed 


Teachers seek peer support and 
interaction more often 


Practitioners are more research 
engaged 


Strategies 
What is your approach? 


e Using Research Ed conference to 
communicate and engage with issues 
(Arms 8 and 9) 


Online pre-post event peer learning 

sessions to engage, strengthen and 

embed learning into practice (Arm 9) 
What strategies and tools will you use? 

e Marketing and engagement 
strategies, CPD offer, support 
package, online tools 

What resources will you need? 

e Time, staffing, ICT, core RCT 

materials for engagement. 


Target Groups 


e KS2 teachers, headteachers, schools 


ResearchEd: ToC — based on 


engagement and research use 


ResearchEd Theory of Change — based on engagement and research use 


Passive trial: 2,500 schools se email message providing URL for ResearchEd conference website 
including dags, all materials POST CONFERENCE 


Participant consults information for personal interest 


Active trial: e 

60 schools (120 staff, ideall 
- invite to conference 

- attend conference 


- access to conference 
- may /may not take part in social media discussions but won't be notified of these unlike AA arm 


Active arm 


Minimal 
effect 


Greatest 
effect 
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Appendix E: Randomisation syntax 


title 'EEFA: Active Trial’. 
set printoback=on. 


GET DATA /TYPE=XLSX 
/FILE='K\EEFA\FINAL list for active trial stats.xlsx' 
/SHEET=name 'FINAL main sample list 221214' 
/CELLRANGE=tull 
/READNAMES=on 
/ASSUMEDSTRWIDTH=32767. 


add value labels region 1 'North' 2 'South’. 


* Check for duplicate nfer_no. 
sort cases by nfer_no. 
match files file=*/first=f1/last=I1/by nfer_no. 


cross f1 by I1. 


* Check for duplicate contact_id. 

sort cases by contact_id. 

match files file=*/first=f2/last=l2/by contact_id. 
cross f2 by I2. 


* Stratify by region. 
freq region. 


sort cases by region. 
* Randomise within the NORTH. 


select if (region=1). 

* Set mtindex to a specific number rather than 'random' so that it can be reproduced 
* change this number for repeated randomisations. 

set rng=mt, mtindex=8. 

compute randome=rv.uniform(0,1). 


print formats random (F15.13). 
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exe. 
sort cases by random. 


list vars random/cases=from 1 to 20. 


numeric group (F2.0). 

*To maintain balanced control group between north and south. 
*North/south split varies between 30/30 and 29/31. 

*See k:\eefa\cfs\activeprovider_order.sps for which provider gets which split. 
if $casenum le 29 group=1. 

if $¢asenum ge 30 and $casenum le 59 group=2. 

if $¢asenum ge 60 and $casenum le 89 group=3. 

if $casenum ge 90 and $casenum le 118 group=4. 

if $¢asenum ge 119 and $casenum le 147 group=5. 

if $casenum ge 148 and $casenum le 176 group=6. 

if $¢asenum ge 177 and $casenum le 206 group=7. 

if $¢asenum ge 207 and $casenum le 235 group=8. 

if $¢asenum ge 236 and $casenum le 264 group=9. 


if $casenum ge 265 group=10. 


freq group. 
sort cases by group. 


save outfile='K:\EEFA\Temp\north.sav'. 


* Randomise within the SOUTH. 


GET DATA /TYPE=XLSX 
/FILE='"K\EEFA\FINAL list for active trial stats.xlsx' 
/SHEET=name 'FINAL main sample list 221214' 
/CELLRANGE=tull 
/READNAMES=on 
/ASSUMEDSTRWIDTH=32767. 


add value labels region 1 'North' 2 'South’. 


select if (region=2). 


set rng=mt, mtindex=20. 


compute randome=rv.uniform(0,1). 
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print formats random (F15.13). 
exe. 
sort cases by random. 


list vars random/cases=from 1 to 20. 


numeric group (F2.0). 

“To maintain balanced control group between north and south. 
*North/south split varies between 30/30 and 29/31. 

*See k:\eefa\cfs\activeprovider_order.sps for which provider gets which split. 
if $casenum le 31 group=1. 

if $¢asenum ge 32 and $casenum le 61 group=2. 

if $¢asenum ge 62 and $casenum le 91 group=s3. 

if $casenum ge 92 and $casenum le 122 group=4. 

if $casenum ge 123 and $casenum le 153 group=5. 

if $casenum ge 154 and $casenum le 184 group=6. 

if $¢asenum ge 185 and $casenum le 214 group=7. 

if $¢asenum ge 215 and $casenum le 245 group=8. 

if $casenum ge 246 and $casenum le 276 group=9. 


if $¢asenum ge 277 group=10. 


freq group. 
sort cases by group. 


save outfile='K:\EEFA\Temp\south.sav'. 


add files file="K:\EEFA\Temp\north.sav‘/in=innorth/file='K:\EEFA\Temp\south.sav'/in=insouth/by 


* Labels as in 'Template for providers 191214’. 


add value labels group 1 '1 - IEE materials' 2 '2 - IEE evidence fair’ 3 


4'4 - Cfl support and website’ 5 '5 - CEM booklet’ 
and light CPD' 


7 '7 - CEM booklet and advanced CPD' 8 '8 - NatCen conference’ 


6 


Cfl 


6 


website 


CEM 


9 '9 - Natcen conference and online community’ 10 '10 - Control group’. 


cross group by region/cells=count col. 


SAVE TRANSLATE OUTFILE='K:\EEFA\Active trial randomisation.xlsx' 
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/TYPE=XLSX 
/VERSION=12 
/MAP 

/REPLACE 
/FIELDNAMES 
/CELLS=VALUES 


/KEEP=contact_id nfer_no region group. 
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Appendix F: Literacy Octopus Engagement Categories 


Analysis was conducted on schools’ engagement in the Literacy Octopus. Whilst the quantitative 
and qualitative characteristics of each level of engagement were specific to each trial arm, we 
adopted an overall consistent approach to categorisation, namely: 


Level of engagement DY =X-Yoq glo) d(o) amo) m=1aler-le(=1nil-yalm-(erere)celiave mom Colalixolalalemlarcelauur-lirelam(iuip) 
data 

None No engagement 

Low Engaged less than expected for that arm 

Mid Engaged as expected for that arm 

High Engaged more than expected for that arm 


Each provider adopted a pre-agree MI data tool, with pre-specified fields in order to capture the 
qualitative and quantitative data required. Providers submitted this data at three time-points (April 
2015, July 2015 and April 2016). NFER researchers carried out the engagement categorisation at 
school level. Some qualitative judgements were made by NFER coders where quantitative data 
was borderline in the categories below. Some double-coding was undertaken on the initial 
datasets in April 2015, to quality assure the consistency of applying the coding schemes. The 
detailed engagement schemes were as follows: 


Arm 1: IEE Passive 


Level of engagement Description 

None 0% materials opened (incl. bounced back after several attempts / unable 
to send) 

Low At least one teacher opened a small amount of materials (e.g. 1 — 20% of 
materials) 

Mid At least one teacher opened some materials (e.g. 21-50% materials) 

High At least one teacher opened most materials (e.g. more than 50% of 
materials) 


NB — MI data for this arm was collated at an individual teacher level. NFER applied the 
above aggregated metric at school level (i.e. one engagement code per school). 


Arm 2: IEE Active 


Level of engagement Description 

None Did not attend Evidence Fair; 0% materials opened (incl. bounced back 
after several attempts / unable to send) 

Low Did not attend Evidence Fair; at least one teacher opened a small 
amount of material (e.g. 1 — 20% of materials) 

Mid At least one teacher attended Evidence Fair; and at least one teacher 
opened some materials (e.g. 21-50% of materials) 

High More than one teacher attended Evidence Fair (NB — where only one 
teacher was listed, one teacher attended Evidence Fair); and at least one 
teacher opened most materials (e.g. more than 50% of materials) 


NB — MI data for this arm was collated at an individual teacher level. NFER applied the 
above aggregated metric at school level (i.e. one engagement code per school). 
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Arm 3: Teaching How2s 
Passive 


Level of engagement 


Description 


None 


No activity on How2s site (i.e. zero techniques and/or nudges recorded) 


Low Small number of active users (e.g. 1 — 3), and limited activity on How2s 
site (e.g. 1 — 15 techniques recorded) 

Mid Small number of active users (e.g. 2 — 5), and moderate activity on 
Howgs site (e.g. 15 — 40 techniques recorded); or larger number of active 
users (i.e. 6 — 25 users) and limited/moderate activity on How2s site (e.g. 
10 — 20 techniques recorded) 

High Small number of active users (e.g. 2 — 5), and high activity on Howes site 


(e.g. over 40 techniques recorded, and in some cases nudges recorded 
also); or large number of active users (e.g. 10+) and high activity on 
Howes site (e.g. over 40 techniques recorded, and in some cases 
nudges recorded also) 


Arm 4: Teaching How2s 
Active 


Level of engagement 


Description 


None 


Did not attend induction; no activity on How2@s site (i.e. zero techniques 
and/or nudges recorded) 


Low 


Did not attend induction; small number of active users (e.g. 1 — 3), and 
limited activity on How2s site (e.g. 1 — 15 techniques recorded) 


Mid 


Attended induction; small number of active users (e.g. 2 — 5), and 
moderate activity on How2s site (e.g. 15 — 40 techniques recorded); or 
larger number of active users (i.e. 6 — 25 users) and limited/moderate 
activity on How@s site (e.g. 10 — 20 techniques recorded) 


High 


Attended induction; small number of active users (e.g. 2 — 5), and high 
activity on How@s site (e.g. over 40 techniques recorded, and in some 
cases nudges recorded also); or large number of active users (e.g. 10+) 
and high activity on How2s site (e.g. over 40 techniques recorded, and in 
some cases nudges recorded also) 


Arm 5: CEM Passive 


Level of engagement 


Description 


None 


Did not respond to initial contact, did not engage with materials (none 
downloaded) 


Low Responded to initial contact, did not engage with materials and did not 
ask for further materials 

Mid Responded to initial contact, engaged with materials 

High Responded to initial contact, engaged with materials, and engaged 


further proactively with provider 


Arm 6: CEM Active 
Light 


Level of engagement 


Description 


None 


Did not respond to initial contact, and no engagement with materials; or 
engaged with initial contact, but no engagement with materials 


Low Responded to initial contact, and engaged with materials, did not attend 
twilight CPD 

Mid Responded to initial contact, engaged with materials, and attended 
twilight CPD 

High Responded to initial contact, engaged with materials, attended twilight 


CPD, and engaged further proactively with provider 
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Arm 7: CEM Active 


Level of engagement 


Description 


None 


Did not respond to initial contact, and no engagement with materials; or 
engaged with initial contact, but no engagement with materials 


Low Responded to initial contact, and engaged with materials, and attended 
the first round of twilight CPD only (i.e. CPD1) 

Mid Responded to initial contact, engaged with materials, and attended two of 
the support activities (i.e. from CPD1, CPD2, INCAS support) 

High Responded to initial contact, engaged with materials, and engaged in all 


three support activities (i.e. CPD1, CPD2, and INCAS). A special note 
was applied if the school also hosted a CPD session to indicate high 
levels of engagement. 


Arm 8: ResearchED 
Passive 


Level of engagement 


Description 


None 


School did not attend conference 


Low School did not attend conference 
Mid One teacher attended conference 
High More than one teacher attended conference 


Note that for Arm 8, the criteria for categorising none and low engagement were the same (i.e. no 
teachers from the school attended the conference. For analysis purposes, these were categorised as 


‘none’). 


Arm 9: ResearchED 
Active 


Level of engagement 


Description 


None 


School did not attend conference 


Low One teacher attended conference 

Mid More than one teacher attended conference; or one teacher attended 
conference, and school attended webinar conference 

High More than one teacher attended conference, and school attended 


webinar conference 


The NFER team used the MI engagement data to help identify case studies. In addition, we 
assessed any change in engagement over time in Literacy Octopus activity, and applied a 
qualitative schema. We used this information to further help identify case studies. 


Change in/continued engagement (2016 data compared qualitatively with 2015 data) 
School did not engage in 2015, and did not engage in 2016 


School engaged less than expected in 2015, and in 2016 had no further or limited engagement 


School engaged as expected for their arm in 2015, and in 2016 this continued; OR, school had 
limited engagement in 2015, but in 2016 increased their engagement moderately 


School engaged more than expected for their arm in 2015, and in 2016 continued to engaged highly; 
OR, school engaged moderately in 2015 and in 2016 increased their engagement 
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1. Positive disposition to academic 
research in informing teaching practice 


(Cronbach's Alpha* = 0.83) 


2. Use of academic research to inform 
selection of teaching approaches 


(Cronbach's Alpha = 0.63) 


3. Perception that academic research is 
not useful to teaching 


(Cronbach's Alpha = 0.55) 


encourage use of academic research 
(Cronbach's Alpha = 0.55) 


5. Active engagement with online 
evidence platforms 


(Cronbach's Aloha = 0.69) 


6. Research knowledge 
(Cronbach's Alpha = 0.62) 


4. Perception that own school does not 


Evidence-based Literacy Support — the ‘Literacy Octopus’ Trial 


* How easy they find it to understand academic research 

¢ Information from research plays an important role in informing my/our teaching practice 

¢ | know where to find relevant research that may help to inform teaching methods/practice 

¢ | am able to relate information from research to my context 

| feel confident about analysing information from research 

| use information from research to help me to decide how to implement new approaches in the classroom 


Academic research was important in identifying specific approach and they used CPD based on academic research 
(Combined score based on 3 items) 


¢ The extent to which the decision to adopt an approach was due to it being based on academic research 
¢ The extent to which thev consult academic research (aenerally) 


| do not believe that using information from research will help to improve pupil outcomes 
Information from research conducted elsewhere is of limited value to our school 


My school leaders/governors do not encourage me to use information from research to improve my practice 
Other staff in my school rarely use information from research to inform their teaching practice 


The extent to which they consult online platforms (generally) 
¢ How easy they find it to understand online platforms 


* Score on research findings question 


¢ Score on research methods question 


*Cronbach’s alpha provides a measure of internal consistency, or reliability, of the measure by comparing how each question performs individually with how all the questions perform together. The 
value of alpha increases when the correlations between the items increase, so a Cronbach’s alpha closer to 1 indicates a more reliable measure. Reliability measures reported in this table are 
based on endpoint survey responses from 557 teachers. Note, the reliability of the outcome measures in the Literacy Octopus study is similar to those found in the development of the Research Use 


Survey (Poet e/ a/., 2015, unpublished). 
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Appendix H: Process evaluation achieved sample 


Table H: Process evaluation: Achieved sample of interviews 


1 IEE — Materials 


2 IEE — Evidence 
Fair 
3 Teaching 
How2s — 
Website access 
4 Teaching 3 
How2s — 
Support and 
website access 
5 CEM - Booklet 3 0 0 
6 CEM - Booklet 
and light CPD 
7 CEM - Booklet 
and advanced 
CPD 
8 ResearchEd — 
Conference 
9 ResearchEd — 


Conference and 
online 
community 
10 - Control 
TOTAL 
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Appendix I: Case Study Implementation and Impact Trails 


In order to more fully understand how impacts occur as a result of engaging with evidence-based 
support, we undertook a detailed analysis of four case studies to explore the trail of mechanisms and 
perceived impacts. We used NFER’s impact trail methodology, which provides a rigorous framework 
for interviewing and analysis, to help understand the sequence of impacts and how they came about. 
We selected four case studies for this more detailed analysis based on where teachers reported 
substantial impacts on their practice as a result of participation in the Literacy Octopus intervention 
and to provide examples of impacts arising from participation in both active and passive interventions. 


The case studies focused on the ‘implementation’ stage of the knowledge-mobilisation process. Our 
analysis revealed that outcomes and impacts are achieved where particular mechanisms are in place 
in the transformation (i.e. the evidence-based materials and support given by the Literacy Octopus 
providers) and implementation of evidence (i.e. the implementation of research evidence in practice 
by the Literacy Octopus schools). The case studies provide examples of how these mechanisms have 
been instrumental in bringing about impacts in the four schools. 


Each case study is accompanied by a visual depiction of the ‘trail’ of impacts occurring in the school; 
from the initial input of the Literacy Octopus provider, to the resultant impacts on teachers’ 
(awareness, understanding and action) and on pupils, and, how these impacts are either facilitated or 
inhibited at each stage by the presence or absence of particular mechanisms. The mechanisms 
referred to in the ‘impact trails’ are explained in Table 12 in the main report. Each case study is based 
on interviews with the key Literacy Octopus contact in the school (usually the literacy coordinator or 
senior leader), a school leader and a class teacher wherever possible. 


Case study A: The headteacher received regular emailed research briefing materials from the IEE. 
Collaborative and embedded approaches to the implementation of this resource were particularly 
prominent facilitating mechanisms. The materials were printed and distributed to all staff in staff 
meetings. At staff meetings, colleagues would discuss the materials and identify strategies that 
appeared relevant to the school context to explore further, and implement in practice as appropriate. 
Tangible impacts on practice were, however, rather limited as it was felt that whilst the materials 
provided a useful ‘kick-start’ of evidence-based ideas, they required substantial further investigation 
and development to affect practice. Nevertheless, the use of different sources of research evidence 
became embedded as part of the schools ongoing school improvement plan. The ‘impact trail’ for 
case study A is presented in Appendix I. 


Case study F: The Literacy Coordinator and headteacher received free access to the Teaching 
How2s website and attended an introductory event on how to use the resource. The subsequent use 
of the resource in practice across the school was particularly facilitated by a planned and phased 
approach to implementation, including a designated member of staff to lead the process; internal staff 
training to familiarise colleagues with the resource; and the trialling of particularly relevant strategies 
across the whole school, staff groups and by individual staff as part of performance management. 
Interviewees identified that the resource had subtle impacts across a wide range of practices and 
numerous members of staff. The ‘impact trail’ for case study F is presented in Appendix I. 


Case study H: A Year 5 teacher (who had since left the school) attended CEM CPD sessions and 
received the accompanying booklet of Key Stage 2 literacy strategies. The school also received 
posters each month with details of literacy activities for pupils. The school effectively mitigated the 
effects of losing the original member of staff who received the CPD by running internal training for all 
school staff to enable the Year 5 teacher to share the learning with colleagues, and identified another 
member of staff to ‘champion’ the implementation of the strategies in practice. Interviewees were well 
informed about the evidence on how to implement particular literacy strategies effectively and were 
convinced by the explicit, yet summarised, evidence that the strategies were worthy of trialling. Senior 
leaders of the school supported the implementation and adaptation of numerous literacy strategies 
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across the whole school, leading to substantial impacts on practice and perceived positive impacts on 
pupils. The ‘impact trail’ for case study H is presented in Appendix I. 


Case study L: Five members of staff (including senior leaders) attended a Saturday ResearchEd 
conference. This case appears to be atypical in this regard as monitoring information shows that 
typically only one or sometimes two members of staff from a school attended the conference. The 
opportunity to attend the conference was opened up to all school staff. Group attendance facilitated 
professional discussion and reflection on practice, and enabled staff to identify, collaboratively, the 
particular ideas from the conference to pursue. Also important was the focus of the conference 
sessions on the practical implementation of literacy strategies, including examples from schools. 
However, this was insufficient information alone to support effective implementation, and the school 
sought further external training on the approach of Reciprocal Reading. An idea inspired by attending 
a conference was thus successfully developed with further input to achieve considerable impact in a 
particular area of literacy practice across the whole school. The ‘impact trail’ for case study L is 
presented in Appendix I. 


The mechanisms referred to in the diagrams are explained in Table 12 of the main report. A ‘+’ 
denotes an enabling mechanism and a ‘-‘ denotes an inhibiting mechanism. Mechanisms relating 
mainly to evidence ‘transformation’ (i.e. the Literacy Octopus provider) are coloured green and 
mechanisms relating mainly to evidence ‘implementation’ (i.e. the Literacy Octopus school) are 
shaded red. Impacts were identified in three main areas: teacher awareness and understanding 
(shaded peach boxes); teacher action/practice (shaded blue boxes); and pupil impact (shaded purple 
boxes). 
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+ relevance e.g. 
existing school culture 
of research 
engagement and use 
of research in school 
improvement 


+ relevance e.g. SLT 
endorse and 
encourage staff to 
engage with research 
evidence 


+ collaborative e.g. 
discussion of 
articles/topics at staff 
meetings and 
decision making on 
implementation 


+ relevance e.g. SLT 
endorse focus on 
strategies 


+ relevance e.g. 
content relevant to 
school’s needs 


+ accessibility e.g. 
accessible style of 
delivery 


+ Status of evidence 
e.g. research 
evidence provides 
validity to the 
strategies and ideas 


Education Endowment Foundation 


Teacher 
awareness of 
practice e.g. 
evidence-based 
strategies and 
practices 


Teacher 
awareness and 
understanding of 
research 
evidence e.g. of 
the range of topics 
being investigated 


+ further enquiry and 
follow-up e.g. regular 
input of 
information/materials 
from IEE focuses 
attention 


+ embedded e.g. part 
of regular CPD and 
school improvement 
plan to read and 
discuss selected 
articles 


+ relevance e.g. 
interested in Education 
Endowment 
Foundation (EEF) 
guidance on effective 
deployment of 
Teaching Assistants 
(TAs) 


+ further enquiry and 
follow-up e.g. seeking 
further training, 
expertise, input on 
particular topics of 
interest 


Teacher practice 
e.g. reviewed 
deployment of 
Teaching 
Assistants 


Teacher research 
engagement e.g. 
interested in 
continuing a 
subscription to the 
materials 


Teacher research 
engagement e.g. 
involvement in 
other research 
studies as 
participants 


+ tryout and review e.g. 
re-deploying TAs to 
provide targeted, earlier 
intervention for pupils not 
achieving expected levels 
of progress 


Teacher practice e.g. 
quicker and earlier 
interventions for 
children delivered by 
TAs 


- accessibility e.g. 
continuing to subscribe 
to the materials will 
depend on affordability 
in the context of other 
expenditure and 
priorities 
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CSF: Access to Teaching How2s website, online materials and face-to-face induction support 


+ relevance e.g. 
breadth of 
teaching strategies 


+ relevance e.g. 
resource shows 
‘how’ to implement 
strategies 


+ status of 
evidence e.g. 
case studies of 
how other schools 
have used How2s 
effectively 


+ Status of 
evidence e.g. 
training explained 
that strategies are 
underpinned by 
research-evidence 


+ collaborative 
e.g. Headteacher 
and acting Deputy 
Headteacher 
attended training 


+ relevance e.g. 
senior staff 
convinced of value 
of using How2s to 
support teachers’ 
professional 
development 


Education Endowment Foundation 


Teacher 
awareness of 
practice e.g. 
classroom 
management, 
mind mapping, 
pupil self- 
assessment 


Teacher 
understanding of 
practice e.g. how 
to use/navigate the 
Howe2s resource 


Teacher research 
engagement e.g. 
interest in 
evaluating the 
impact of 
strategies and 
participating in 
research 


+ further enquiry 
and follow-up e.g. 
exploring range of 
How2s resources 


+ planned e.g. 
action plan for 
implementing use of 
Howes, including 
costs, resources, 
staffing, review. 


+ collaborative e.g. 
staff meeting to 
introduce How2s 
resource 


+ further enquiry 
and follow-up e.g. 
staff encouraged to 
initially explore 
resources 


+ relevance e.g. 
required strategies 
to address a pupil’s 
disruptive behaviour 


- collaborative e.g. 
reluctance of some 
staff to engage with 
the resource 


Teacher 
practices e.g. 
teacher 
implemented 
behaviour 
strategies 


Pupil behaviour 
e.g. reduction in 
a pupil’s 
disruptive 
behaviour and 
improved 
engagement in 
lessons 


+ champion e.g. 
acting Deputy 
Head leads whole- 
school 
implementation of 
new pupil self- 
assessment 
strategy 


+ tryout and 
review e.g. acting 
Deputy Head 
trialled pupil self- 
assessment 
strategy and 
reviewed ina 
subsequent staff 
meeting 


+ collaborative 
e.g. staff meeting 
to introduce pupil 
self-assessment 
strategy 


+ embedded e.g. 
approach to KS2 
pupil self- 


assessment written 


into assessment 
policy 
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Teacher practices 

e.g. consistent and 

simple approach for 
pupils to explicitly 


self-assess their own 


work across all 
subject areas 


Teacher practices 
e.g. use of mind 


Mapping approaches 


to break down 
different stages of 
planning and 
encourage pupils to 
work independently 
in developing their 
extended writing 


Pupil impact e.g. 
pupils producing 
more detailed and 
longer pieces of 
writing using mind- 


mapping techniques 


+ tryout and review 
e.g. staff encouraged 
to select one new 
teaching strategy per 
half term to trial and 
review with colleagues 
in phase meetings 


+ embedded e.g. use 
of How2 resources to 
address teacher needs 
identified through 
Performance 
Management and 
lesson observations 


+ further enquiry and 
follow-up e.g. 
regularly updated 
content on How2s 
resource 


+ collaborative e.g. 
staff send each other 
‘nudges’ to highlight 
particular How2 
resources of interest 


+ adaptation e.g. 
adapted ideas on mind 
mapping to help 
younger pupils 
condense, organise 
and summarise 
information 


+ relevance e.g. used 
How2 visuals to help 
pupils understand the 


concept of mind 
mapping 


+ relevance e.g. all 
staff able to access the 
resource and visuals to 
show ‘how’ to 
implement strategies 


Teacher practices 
e.g. useful 
strategies for new 
teachers and 
reminders of 
effective strategies 
for longer-standing 
teachers to raise 
the quality of 
teaching 


Teacher practices 
e.g. questioning of 
pupils; class 
management 
through 
establishing 
routines; liaising 
with parents about 
homework 


Pupil impact e.g. 
improved behaviour 
due to clear and 
consistent 
classroom routines 


- accessibility e.g. 
leaders will 
evaluate the 
relative impact of 
How2s compared 
with other 
professional 
development 
resources in 
deciding whether 
or not to continue 
the subscription 


+ collaborative 
e.g. the school is 
keen to become a 
‘hub’; supporting 
and training other 
local schools to 
use the Teaching 
Howes. 
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+ tryout and review 
e.g. staff fed back on 
their experiences of 


Teacher + champion e.g. Deputy 
awareness and head led implementation of 
understanding of strategies 


Teacher practice e.g. 
more consistent literacy 
teaching approaches 


+ relevance e.g. Teacher 
literacy identified as awareness and 
an area of school understanding of 


Teacher practice e.g. 
buddy reading 
implemented with all 


+ Champion e.g. 
teacher enthusiasm 


+ relevance e.g. SLT 
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improvement 


+ status of evidence 


e.g. detailed 
discussion of 


evidence-base for 
strategies and the 


conditions for 
effective 


implementation in 


training 


+ collaborative e.g. 
opportunity for face- 
to-face and in-depth 
discussion in training 


Education Endowment Foundation 


practice e.g. 
various evidence- 
based literacy 


strategies 


identified literacy 
strategies to trial 
across the school 


+ collaborative e.g. 
in-house staff training 
to disseminate 
selected literacy 
strategies 


- champion e.g. 
teacher who attended 
CPD left the school 


practice e.g. 
various evidence- 
based literacy 
strategies 


+ Status of evidence e.g. 
used evidence to justify 
implementation of 
strategies 


+ tryout and review e.g. 
implemented buddy 
reading; spelling games; 
and mind-mapping 
strategies across the 
school 


+ adaptation e.g. adapted 
strategies for KS1 pupils 


+ further enquiry and 
follow-up e.g. ongoing 
input from provider giving 
prompts and ideas and 
responding to queries 


- status of evidence e.g. 
where evidence not clear 
unable to provide rationale 
for introducing strategies 


+ status of evidence e.g. 
implemented buddy 
reading as evidence 
suggested most effective; 
with two-year age gap in 
buddy pairs, where 
possible 


year groups 


Teacher practice e.g. 
used spelling 
strategies with all year 
groups 


Teacher practice e.g. 
used posters to focus 
literacy activities, 
including ‘word of the 
week’ 


Teacher practice e.g. 
used mind mapping 
strategies 


Teacher practice e.g. 
used ‘magpie’ books 


implementing 
strategies at staff 
meetings; positive 
feedback so decided to 
continue 


+ collaborative e.g. 
whole school 
implementation 
engages all staff and 
staff can share 
challenges and 
successes and engage 
in professional 
discussion 


+ adaptation e.g. to 
address logistical 

challenges identified 
and refine approach 


+ embedded e.g. 
plans to continue to 
use buddy reading and 
other literacy strategies 
long term 


and enhanced quality 
of literacy teaching 


Teacher research 
engagement e.g. more 
likely to seek evidence- 
based strategies in the 
future as positive 
experience 


CPD practice e.g. 
gained experience of 
implementing whole- 
school strategies which 
will apply to other 
aspects of practice 


Pupil impact e.g. 
raised pupil 
achievement in reading 


Pupil impact e.g. 
enhanced reading 
comprehension skills 
through more 
discussion of books 
and words 


Pupil impact e.g. 
enhanced confidence 
reading to peers in 
non-threatening ‘buddy 
reading’ context 


Pupil impact e.g. 
pupils making friends 
across age-groups 
through buddy reading 
Pupil impact e.g. 
extended vocabulary 
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+ relevance e.g. 
inspiring practical 
ideas and 
demonstrations from 
other schools 


+ status of evidence 
e.g. explicit citations 
to evidence 


+ collaborative e.g. 
several school staff 
attended allowing 
professional reflection 
and discussion 


+ relevance e.g. 
senior staff keen to 
develop evidence- 
informed practice 


ducation Endowment Foundation 


Teacher 
awareness and 
understanding of 
practices e.g. of 
reciprocal reading 


Teacher 
awareness and 
understanding of 
literacy strategies 
e.g. slow writing 


Teacher 
awareness and 
understanding of 
research 
evidence e.g. how 
the EEF toolkit 
works 


+ relevance e.g. senior 
staff identified 
reciprocal reading as 
strategy to improve 
‘guided reading’ 


+ collaborative e.g. 
staff meeting and 
informal discussion to 
disseminate information 
on reciprocal reading 


+ further enquiry and 
follow-up e.g. further 
enquiry into a broader 
range of evidence on 
reciprocal reading 


+ adaptation e.g. staff 
adapted their existing 
guided reading 
practices to incorporate 
elements of a 
reciprocal reading 
approach with their 
class 


+ champion e.g. 
teacher enthusiasm 


+ relevance e.g. 
teacher’s interest in the 
potential of slow writing 
as a strategy to support 
struggling writers 


+ collaborative e.g. 
professional discussion 
of evidence 


Teacher literacy 
practices e.g. 
several staff 
implemented a 
reciprocal reading 
approach with their 
class 


Teacher literacy 
practices e.g. a 
teacher 
implemented 
elements of ‘slow 
writing’ in their own 
class with struggling 
writers 


Teacher research 
engagement e.g. 
more use of EEF 
toolkit; greater 
interest in evidence- 
informed practice; 
developing interest 
in action research 
(attended further 
conference) 


+ tryout and review e.g. 
informal review of how staff 
experienced reciprocal reading 
which highlighted benefits but 
also practical challenges of 
implementation 


- relevance e.g. insufficient 
information from conference on 
detail of ‘how’ to implement 
reciprocal reading effectively 


+ planned e.g. action plan for 
implementing reciprocal reading 
across the school 


+ further enquiry and follow- 
up e.g. reading lead and senior 
leader attended external 
training on reciprocal reading 
and obtained resources 


+ collaborative e.g. internal 
training for all staff on reciprocal 
reading 


+ champion e.g. reading lead 
provided ongoing coaching 
support to colleagues to 
implement reciprocal reading 
effectively 


+ embedded e.g. reciprocal 
reading approach incorporated 
into whole school literacy policy 
and all staff required to use the 
approach 


- collaborative e.g. teacher 
using slow writing left the 
school 


- relevance e.g. slow writing 
approach did not align with 
leader's aspiration to improve 
children’s independent writing. 


- status of evidence e.g. 
senior leader’s perception of a 
lack of evidence on the 
effectiveness of slow writing 


Teacher knowledge 
and practice e.g. 
greater depth of 
knowledge of 
reciprocal reading 
pedagogy; more 
reflective practice; 
more confident to try 
new approaches 


Teacher practices 
e.g. all staff using 
reciprocal reading 
approach 


Pupil literacy skills 


e.g. more child-led (as 
opposed to teacher- 
led) group discussion 
of texts; children using 
wider range of 
dialogue and more 
confident to discuss 
texts in group reading 
activities 


+ further enquiry 
and follow-up e.g. 
planned further 
training for all staff 
(and TAs) with 
external provider to 
consolidate 
understanding of 
reciprocal reading 
approach and refine 
practice 


+ embedded e.g. 
planned ongoing 
internal training for 
new staff on the key 
features of effective 
reciprocal reading 


+ embedded e.g. 
planned review to 
refine reciprocal 
reading practice to 
ensure effectiveness, 
differentiation and 
consistency across 
the whole school 


Pupil literacy 
skills 


e.g. anticipated 
long-term impact 
on children’s 
reading 
performance 
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